NVIDIA Nemotron 3.5 Super VL joins AWS for multimodal AI tasks

A multimodal leap for agents

NVIDIA has introduced the Nemotron 3.5 Super VL, a multimodal model designed to power agentic AI workflows. These workflows are systems where AI can perform multi-step tasks by reasoning through problems and using external tools. This specific model is a mixture-of-experts (MoE) architecture, which means it only activates a portion of its total parameters during any single calculation.

The model contains 120B total parameters but only activates 12B parameters per token. This design allows the system to maintain the high memory capacity of a large model while keeping the actual computation costs similar to a much smaller model. It also features a context length of up to 1M tokens, allowing it to process very long strings of information at once.

Because it is a multimodal model, it can understand and process different types of data including text, images, documents, and video. This versatility allows a single model to serve multiple roles, such as analyzing a video clip for incident reports or reading a complex table in a document. NVIDIA has released the weights and training recipes, giving developers the freedom to customize the model for specific needs.

Hybrid architecture and efficiency

The Nemotron 3.5 Super VL uses a unique hybrid architecture that combines Mamba-2, attention, and expert layers. Mamba-2 is a type of architecture that carries a constant-size state during generation. This is a significant technical distinction because it prevents the memory requirements from growing as the sequence length increases, making long contexts more affordable to run.

The model also incorporates a RADIO vision encoder to handle visual inputs. To improve speed, it uses a recursively shared Multi-Token Prediction (MTP) head. This head supports native speculative decoding, a technique where the model predicts multiple future tokens at once to speed up the generation process. These architectural choices aim to balance high-quality reasoning with efficient hardware utilization.

NVIDIA has already validated this architecture on high-end hardware, including the p6-b200.48xlarge and p5en.48xlarge instances. These instances utilize NVIDIA B200 and H200 GPUs, respectively. Testing shows that the model can achieve specific performance targets, such as a time to first token (TTFT) under 600 ms and an inter-token latency (ITL) under 100 ms per token, depending on the deployment topology.

Deployment on Amazon infrastructure

Developers can deploy the Nemotron 3.5 Super VL on several Amazon Web Services (AWS) platforms, including Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS. EKS Auto Mode is particularly useful because it allows AWS to manage the scaling, GPU support, and Elastic Fabric Adapter (EFA) support automatically. The Kubernetes manifests remain the same across these different orchestrator options, simplifying the transition between environments.

The deployment architecture separates the control plane from the data plane. In this setup, CPU nodes handle system and test pods, while dedicated GPU nodes run the actual model. The model weights are stored in a shared Amazon FSx for Lustre volume, and the container images are hosted in the Amazon Elastic Container Registry (Amazon ECR). This structure ensures that the heavy lifting of model inference is isolated to the necessary hardware.

For vision-language models like Nemotron 3.5 Super VL, the image and video encoding stage can be disaggregated. This means the encoding can run on its own separate node rather than on the same node as the main inference. This flexibility helps engineers optimize for different performance needs, such as prioritizing throughput for many users or minimizing latency for a single interactive session.

Real-world agentic use cases

The model is specifically tuned to support complex agentic workflows that require both reasoning and visual understanding. In the coding space, it can be used to test and validate code, analyze errors, and recommend specific fixes. For web research, it can cross-reference and verify various sources that are returned by search tools, helping to ensure the accuracy of gathered information.

In the realm of document intelligence, the model can read tables, charts, and various page layouts to produce concise summaries. This is particularly useful for processing financial reports or technical manuals where the layout is as important as the text. It also has applications in cybersecurity, where it can identify vulnerabilities and triage various threats based on provided data.

Furthermore, the model excels at video understanding tasks such as searching for specific content, reviewing footage, and performing incident analysis. It integrates with major open agent frameworks through NVIDIA reference implementations and the NVIDIA Agent Toolkit. These tools provide developers with the necessary blueprints to build production-ready applications that can see, hear, and reason about the world.

Measuring and scaling performance

AWS provides specific tools to help developers validate their clusters and measure model performance. The NVIDIA Collective Communications Library (NCCL) all-reduce micro-benchmark can be used to verify that the cluster and network are working correctly. By checking for utilization on all GPUs and traffic on all EFA adapters, engineers can confirm the underlying infrastructure is healthy before deploying the model.

To measure actual inference performance, users can use the NVIDIA aiperf tool. This tool provides key performance indicators such as time to first token (TTFT), inter-token latency (ITL), and throughput in tokens per second (TPS). These metrics are essential for planning capacity because they show how latency and throughput change as the number of concurrent requests increases.

By plotting these metrics, developers can draw a latency budget to see how many requests a single replica can serve. For example, if a service requires a TTFT under 1 second, the chart helps identify the maximum throughput possible within that limit. If the inter-token latency rises, it signals a need for more decode capacity, while a rising time to first token suggests a need for more prefill capacity.