NVIDIA adds AI workload tracking to Kubernetes clusters

The visibility gap in AI

Deploying AI workloads at scale often creates a visibility problem for platform teams. While a platform might have a set configuration, application teams often update their specific workloads independently of the underlying infrastructure recipe.

This means a reproducible deployment bundle is not always enough to tell an administrator which specific code or model is running on a node. The problem is worsened by mutable image tags, such as :latest, which can result in different nodes running different code under the same name.

As developer.nvidia.com reported, NVIDIA is addressing this by adding a qualified AI bill of materials (AIBOM) component to the NVIDIA AI Cluster Runtime (AICR). This tool helps close the gap between infrastructure configuration and actual workload execution.

How AICR and AIBOM work

The two components solve different parts of the same problem. AICR focuses on the deployment layer, providing version-locked recipes for drivers, operators, and Kubernetes components. It ensures that the GPU-accelerated platform itself is validated and reproducible.

In contrast, k8s-aibom sits higher up the stack and watches the Kubernetes workload and pod APIs. It emits a CycloneDX 1.6 AIBOM for every recognized AI workload it detects. It specifically looks for signals like native Dynamo graphs, NIMService, and CronJob resources.

When these tools work together, the system can automatically classify a workload's category and serving runtime. For example, if a user deploys a vLLM instance, the AIBOM can identify the specific model being served, such as Qwen, and the container image being used.

Tracking models and dependencies

The AIBOM provides a detailed breakdown of the AI stack rather than just listing a container image. It identifies specific components like inference runtimes, agents, and Retrieval-Augmented Generation (RAG) components. It also tracks external AI APIs and vector databases used by the applications.

One of the primary benefits for administrators is the ability to detect changes or drift in AI dependencies. The system provides four confidence tiers for these attributes: declared, inferred, unresolved, or verified. This allows reviewers to see exactly how the system reached its conclusion about a specific model or runtime.

The tool also helps with security reviews and incident response by finding unregistered or shadow AI workloads. If a namespace is opted into the inventory, the controller automatically records the workload specifications even before the pods are fully running.

Performance and resource impact

Because the system is designed for large-scale production, NVIDIA tested the overhead of the k8s-aibom controller on Google Kubernetes Engine (GKE). The tests were conducted on CPU-only nodes to measure the baseline impact of the inventorying process.

The results showed that the controller is highly efficient, using an average of 2.2 millicores of CPU. It also maintained a memory working set of between 28 and 34 MiB. These figures represent the steady-state observation of the controller's overhead.

During operation, the controller made approximately four API-server requests per minute. It also triggers a regeneration of the AIBOM only when the tested workload's image tag changes, ensuring that the tracking process does not unnecessarily strain the cluster's resources.

Future integration paths

While the current integration provides a powerful inventory, there are several areas for future development. Currently, the runtime AIBOM is not signed, meaning the observations are not yet tamper-evident. Adding signing would allow for more rigorous verification of the captured data.

NVIDIA also plans to explore connecting verified provenance to the runtime inventory. This would involve checking Sigstore signatures on model claims against configured trust roots. This would allow the system to assign 'verified' status to claims based on established identities.

Finally, there is a goal to correlate the AICR deployment intent with the observed workloads. This would involve creating a shared identity between the infrastructure and the workload coverage. Currently, the AIBOM tells you what was declared and what was found, but it does not yet prove successful execution.

Implementation details

To use these tools, teams can start with a GKE cluster running Kubernetes 1.32 or later with NVIDIA H100 GPU nodes. The workflow involves generating a recipe via the AICR CLI, which selects the k8s-aibom component as part of the standard configuration.

Once the recipe is rendered into a bundle and deployed, the k8s-aibom controller runs in a dedicated namespace. Users can then query the inventory to see the status of their workloads. The system provides a clear view of whether a workload is ready, synced, or if any sink failures have occurred.

The AIBOM document is stored inline on the resource when it is small enough. Users can export this data to a JSON file and use tools like jq to view the specific components. This provides a transparent way to audit the entire machine-learning-model and container stack.