GPUWorks open-sources lightweight AI inference platform

Unified hardware management

GPUWorks has released its enterprise AI inference platform under the Apache-2.0 open-source license. The tool is designed to help organizations manage a diverse pool of hardware accelerators from various manufacturers. By pulling these different chips into one managed system, companies can deploy models more consistently across their infrastructure.

The platform supports a wide range of hardware, including NVIDIA and AMD chips alongside Huawei Ascend, T-Head PPU, and Hygon DCU. It can handle various environments, such as bare-metal servers, virtual machines, and cloud containers. This flexibility allows a single cluster to span private data centers and public clouds simultaneously.

As reported by pandaily.com, the system automatically matches specific AI models to the available hardware. It also estimates the memory requirements for each model before deployment. Once configured, the platform serves these models through an API that is compatible with OpenAI standards.

Lightweight deployment and scale

One of the primary goals of the GPUWorks team was to keep the software footprint as small as possible. The entire installer is just over 40MB and can be installed with a single command in approximately 60 seconds. Notably, the platform does not require Kubernetes, which is a common but complex container orchestration system often used in these environments.

For security-conscious organizations, the system supports offline use. Users can import the installer, model files, and runtime images into internal networks without needing an active internet connection. The platform also includes a catalog with more than 1,000 launch configurations for nearly 400 different models.

These configurations bundle everything needed for a specific model and chip combination, including the image, inference engine, and weight versions. To help with troubleshooting, the platform includes a built-in AI assistant to diagnose faults. It also features a scheduler that provides explanations for every placement decision it makes.

Efficient model sharing

A significant technical feature of the platform is its ability to manage how models occupy GPU memory. The system uses a 'level-one sleep' mode that keeps model weights in the host memory. This prevents the engine from having to restart or reload weights from storage every time a model is needed.

This architecture allows several large models to share the same GPUs in turns. In testing, the team found that waking a sleeping DeepSeek-V4.1-Flash deployment took as little as 3.6 seconds. This is a major improvement over a cold start from storage, which can take over 16 minutes for large models on certain hardware.

Administrators can manage these resources using various policies, such as priority levels or scheduled rotations. The platform also supports multi-node parallelism for larger workloads. It can configure specific networking links like RDMA or Ascend HCCS to ensure high-speed communication between different nodes.

Monitoring and maintenance

The platform provides deep visibility into how the AI infrastructure is performing. It tracks specific metrics like token usage and cache hits for every API key and model deployment. It also monitors card-hours per million output tokens and identifies which GPUs are currently sitting idle.

For large-scale operations, the system includes built-in load tests to check throughput and latency before scaling up. This helps ensure that the infrastructure can handle the expected traffic. It also supports prefill-decode disaggregation, which is a method of separating different parts of the AI generation process to improve efficiency.

The software also automates routine maintenance tasks to prevent downtime. During node maintenance, the platform adds new replicas before draining the old ones. If a machine goes offline unexpectedly, the system automatically restores services on another eligible node in the cluster.