NVIDIA deploys AI agents to automate hardware and software failure diagnostics

Automating complex cluster diagnostics

NVIDIA recently unveiled an AI operations framework designed to manage the complexities of DGX SuperPOD environments. These large-scale clusters are used to train massive AI models, but they often face intricate failures that are difficult to pinpoint. The new system helps site reliability engineers trace problems across separate infrastructure layers more efficiently.

The primary goal of this framework is to reduce the manual effort required to follow a fault from start to finish. In traditional setups, an engineer might have to check hardware logs, network signals, and software job statuses individually. This new approach uses AI agents to examine these signals simultaneously to find the root cause of a training failure.

NVIDIA described this system as the AI Factory Operations Agent during an announcement on Wednesday, Nov. 4. It is built to provide engineers with a clear view of why a system is underperforming or failing. The framework presents possible causes and specific recommendations alongside the evidence it gathered for human review.

The NemoClaw blueprint and agent roles

The framework is organized around a specific blueprint called NemoClaw. This blueprint coordinates various agents, each assigned a distinct operational role to prevent overlap in responsibilities. By keeping these roles separate, the system ensures that each agent focuses on a specific part of the infrastructure stack.

The Job Recovery Agent is specifically tasked with investigating failed training jobs to see what went wrong during execution. Meanwhile, a Hardware Agent monitors the physical components, looking for GPU faults, thermal anomalies, and PCIe errors. These agents work in tandem to determine if a software crash was caused by a physical hardware malfunction.

Other specialized agents include an Observability Agent that builds dashboards using Prometheus data and Grafana Loki logs. A Kubernetes agent can read cluster resources and scheduling status, while a Cluster Management Agent uses specific APIs to troubleshoot the overall cluster health. This modular design allows NVIDIA to add new agents or infrastructure integrations without changing existing components.

Communication and technical architecture

To facilitate communication between these different agents and external tools, the framework uses the Model Context Protocol (MCP). This is a standard protocol that allows AI applications to connect with external data sources and tools reliably. Engineers can interact with the entire system through a simple chat interface to see the results of the agents' work.

The application runs within the Kubernetes control plane, which is the system that manages the lifecycle of containerized applications. It hosts the chat interface, the OpenClaw gateway, and the inference endpoint. Helm is used to handle the deployment of these various components into the environment.

The system also utilizes the NVIDIA OpenShell runtime to manage interactions. OpenShell creates a separate Kubernetes sandbox for every shell-command session. This design choice limits the agents' direct access to sensitive Kubernetes credentials while still allowing them to perform necessary diagnostic tasks.

Security and access controls

NVIDIA implemented strict security measures to ensure that the AI agents do not have unrestricted power over the infrastructure. Role-based access controls and server-side inspection tools are used to restrict what the agents can do. These layers of security are designed to prevent the agents from making unauthorized changes to the production environment.

While the agents are capable of reading pod and configmap information, they are explicitly blocked from creating new pods or updating existing ones. This ensures that the agents remain in a diagnostic role rather than an administrative one. The framework keeps evidence available for review so that human engineers can verify every finding before taking action.

In production environments, the agents have already demonstrated their ability to connect findings across workload, hardware, and network layers. By providing a clear audit trail of evidence, the system allows engineers to trust the AI's diagnosis while maintaining human oversight. This balance of automation and control is central to the framework's design for enterprise use.

Deployment and local inference

The framework is designed to be flexible regarding where it is hosted and how it processes information. NVIDIA supports on-premises deployment, which is important for companies that want to keep their data and infrastructure within their own private data centers. This allows for a more controlled environment for sensitive AI training workloads.

For the actual processing of information, the system can use locally served NVIDIA Nemotron open models. Using open models for inference means that the data used for diagnostics does not necessarily have to leave the local network. This supports the privacy requirements of many large-scale enterprise customers who operate DGX SuperPODs.

Because the framework is modular, it can be scaled as a company's infrastructure grows more complex. As more hardware types or software layers are added to the cluster, new agents can be integrated into the NemoClaw blueprint. This ensures the diagnostic tool remains relevant as the underlying AI hardware and software continue to evolve.