NVIDIA releases Attribution tool to diagnose distributed AI failures

The challenge of distributed errors

When training large AI models across a distributed cluster, a single hardware failure can trigger a massive cascade of symptoms. A faulty GPU or a network fabric issue might manifest as communication errors on dozens of other nodes, creating a confusing trail of logs for engineers to sift through.

Often, the error message a user sees only indicates where the workload eventually stopped, not where the problem actually began. This makes it difficult to determine if a failure was caused by a software bug, a network congestion issue, or a physical hardware fault. Identifying the root cause manually in these environments is a significant bottleneck for production scaling.

NVIDIA Attribution addresses this by analyzing failures in the context of a specific workload rather than as isolated node events. By correlating data from various sources, the tool provides a structured diagnosis that includes the outcome, the root cause, affected nodes, and a supporting evidence trail. It helps teams move past the symptoms to find the actual source of the problem.

How the attribution engine works

The system operates through a four-stage pipeline designed to turn raw evidence into a clear verdict. First, the NVIDIA Attribution Blueprint identifies the specific workload attempt using scheduler identifiers or an execution window. It then collects observations from application logs, syslogs, GPU driver messages, and network metrics.

Next, the Attribution Service invokes a core engine to extract and correlate significant events. This engine uses deterministic rules to evaluate the timing and interaction of events like GPU XIDs or scheduler transitions. These rules are designed to separate downstream symptoms—such as communication errors—from the primary event that triggered the failure.

Finally, the system returns a structured result that combines the verdict with the workload outcome and the attributed domain. If multiple causes fit the evidence, the tool marks the sub_domain as ambiguous to reflect uncertainty. This ensures that the diagnosis is based on the quality and completeness of the underlying telemetry rather than forcing a definitive answer from incomplete data.

Architecture and deployment

NVIDIA has split the tool into two main components to allow for flexibility in different environments. The Attribution Service is a self-hosted HTTP wrapper around the core logic that receives normalized observations and returns a single verdict. This allows the core logic to remain consistent even if the underlying data sources change.

The Attribution Blueprint serves as a Python reference orchestrator that connects specific cluster components to the Service. It handles the discovery of workloads, runs the necessary collectors, and writes the final results to a dashboard or database. This separation allows teams to apply the same attribution rules across different clusters while customizing the connectors for their specific schedulers and telemetry backends.

The Blueprint comes with pre-built connectors for common tools like Slurm, Kueue, Loki, and PromQL metrics. Because the system uses standardized contracts for observations and results, developers can easily add new rules or connectors. This modularity means that new integrations will automatically feed into the existing diagnosis pipeline without requiring a rewrite of the core logic.

Real-world impact and use cases

NVIDIA Attribution has already been used in production across its AI infrastructure for several years. It currently analyzes millions of workloads each week across hundreds of thousands of GPUs to maintain system health. This scale allows it to identify complex patterns that would be nearly impossible for humans to spot manually.

In one large RDMA-backed cluster, the tool successfully correlated port errors, connection resets, and scheduler events to pinpoint specific host and switch ports. This allowed the engineering team to target specific fabric repairs and perform precise congestion tuning. By identifying the exact location of the problem, the team could resolve the issue much faster than by investigating the entire cluster.

The tool also provides specific guidance for recovery based on the identified failure mode. For example, if a GPU encounters a hard fault, the system can identify the specific hardware error and suggest the next action for that node. This complements existing readiness checks and continuous health monitoring systems to keep AI workloads running smoothly.

Extending the framework

Because the tool is open-sourced, infrastructure teams can encode specific failure patterns once and apply them across multiple clusters. Developers can write new rules for unique failure modes or implement custom connectors for proprietary telemetry backends. This allows the tool to grow alongside the specific needs of an organization's infrastructure.

The rules engine supports both declarative and Python-based rules, making it accessible to different types of developers. A rule change can be committed and shipped with the Service, ensuring that the logic remains consistent across the fleet. The repository includes a rule-authoring guide to help users register new symptoms and verify the catalog with test cases.

By using the Attribution Blueprint, practitioners can customize the solution to fit their specific environment while still benefiting from NVIDIA's core logic. Downstream systems can then consume the structured output—such as the terminal state and sub_domain—instead of having to parse complex, product-specific logs. This simplifies the integration with incident, recovery, and fleet-management systems.