NVIDIA open-sources tools to detect slow GPU nodes in clusters

The problem of slow nodes

When training large AI models, researchers often use synchronous distributed training. In this setup, every single GPU node must complete its forward and backward passes before the entire cluster can proceed to the next step. This means the speed of the whole job is limited by the slowest individual node in the system.

A major challenge for operators is identifying these problematic nodes. A GPU might not throw a hard error or trigger a standard alert, but it may still deliver performance significantly below its target. NVIDIA argues that simply checking if a GPU is 'active' is a poor way to measure this, as a node can show normal utilization while still underperforming.

To address this, NVIDIA points to a metric called goodput, which measures the volume of work successfully completed. Because scale makes these issues costly, companies need better ways to monitor hardware health. For example, Meta reported 419 unexpected interruptions over 54 days on a 16,384-GPU cluster, with most issues traced back to hardware problems.

How NVCRE certifies clusters

NVIDIA's Cluster Readiness Engine (NVCRE) is a Kubernetes controller designed to exercise the cluster with real workloads. Instead of just checking if software is running, it runs a suite of tests including NCCL communication tests and NeMo Framework pretraining jobs. These tests stress the GPU memory, compute power, and the network fabric used for distributed training.

The engine focuses on communication bus bandwidth as a primary metric. Because results vary based on hardware and fabric, NVCRE does not come with universal pass/fail lines. Operators must choose which nodes to test and define the specific performance thresholds that determine whether a node is considered healthy or suspect.

When a test fails on a large group of nodes, NVCRE automates the process of finding the culprit. It performs a bisection, splitting the failing group into halves and rerunning tests until it identifies the specific suspect nodes. This replaces the manual troubleshooting engineers would otherwise have to perform to find a single faulty component in a massive cluster.

Monitoring and remediation

While NVCRE handles certification, NVSentinel acts as the health monitoring agent. It watches for signals like GPU Xid events, network interface states, and cloud-provider maintenance notices. Unlike NVCRE, it does not run extra workloads to gather data, which keeps the GPU capacity free for actual production jobs.

NVSentinel is designed with a cautious approach to cluster management. It ships in a monitor-only mode, publishing health events as Kubernetes node conditions without taking action. This allows operators to compare the tool's signals against their own knowledge before they decide to enable automated responses like cording, draining, or repairing nodes.

Once operators trust the signals, they can write rules using Common Expression Language to decide which events trigger a response. A circuit breaker is also included to cap how many nodes can be quarantined at once. This prevents a correlated failure from removing too much capacity from the cluster simultaneously, ensuring the remaining hardware can still handle the workload.

Lifecycle and drift management

The NVIDIA AI Cluster Runtime (AICR) handles the initial configuration of the environment. It generates a version-locked recipe based on the specific hardware, operating system, and workload intent. This ensures that all components, such as the GPU Operator and NVSentinel, are installed in the correct order with the correct versions.

AICR also includes a drift-comparison command to monitor the cluster over time. It compares a live snapshot of the cluster against the baseline recipe to detect manual changes or updates applied outside of the standard pipeline. If a team enables the fail-on-drift flag, the system can automatically stop a deployment stage if the cluster configuration has diverged from the intended state.

The tools work together to create a closed loop for hardware health. After NVSentinel identifies and remediates a faulty node, NVCRE runs its certification suite. The node remains gated and out of rotation until it passes these tests. This ensures that repaired nodes or new additions are fully validated before they are allowed to join the production training job.

Open source and adoption

NVIDIA has released these tools under the Apache 2.0 license, which allows organizations to read, modify, and reuse the code. While the license is permissive, the tools still rely heavily on NVIDIA-specific components like DCGM for diagnostics and NCCL for communication. This means that while the code is open, the underlying hardware-specific ecosystem remains tied to NVIDIA.

The design of DSX OS is intended to be a measurement and control kit rather than a push-button enforcement system. Because the defaults are tuned for safety, operators must take active steps to enable full automation. They must first measure their cluster to establish a baseline, then set their own thresholds, and finally switch on the automated response features.

NVIDIA plans to demo these tools at the upcoming KubeCon in Salt Lake City. For teams looking to adopt DSX OS, the primary benefit is a structured way to manage the lifecycle of a GPU cluster. It provides the visibility needed to catch silent performance degradations that would otherwise waste significant time and money during large-scale AI training.