NVIDIA releases GPU memory specs and recipes for Cosmos world models

Deployment recipes for physical worlds

NVIDIA has released new implementation recipes designed to help developers use Cosmos world models in complex simulations. These world models are systems capable of understanding and generating data that mimic physical environments. The new guidance specifically targets three high-stakes industries: vehicle development, robotics, and surgical procedures.

The framework is designed to be versatile, supporting training with a wide variety of inputs including text, images, video, audio, and actions. Developers can choose between full supervised fine-tuning or using LoRA, which is a technique for efficient model adaptation. The system also supports common data formats like JSONL and WebDataset, as well as LeRobot and custom adapters.

According to www.tokenpost.com, these recipes provide the necessary details for model adaptation and testing. This move follows earlier releases of Cosmos data and simulation tools, providing a more complete toolkit for engineers. The goal is to give developers a clear path for deploying these models onto specific hardware configurations.

Simulating autonomous vehicles

For the automotive industry, NVIDIA introduced specific recipes for autonomous vehicle development. One model, Cosmos3-Nano-Transfer-Auto, is designed to generate multi-view video for offline testing. This allows developers to create diverse scenarios without needing to physically drive a car in every situation.

The system can handle input and output video at 30 frames per second, with clips reaching up to 400 frames at resolutions of 480p or 720p. In a reference run, the technology generated seven 720p videos totaling 297 frames. It also produced 99 LiDAR sweeps, which are used by vehicles to map their surroundings using light pulses.

Another model, Cosmos3-Nano-Sim-Auto, focuses on closed-loop evaluation. This means it can generate observations as a driving policy changes or as a human user modifies the scenario. This allows for a dynamic testing environment where the AI can react to a changing world in real time.

Robotics and surgical applications

The robotics recipes focus on translating visual data into actionable scenes. One method converts depth video into RGB scenes while ensuring that the geometry and motion of the objects remain intact. This is essential for robots that need to understand the 3D structure of their surroundings.

In other cases, the model can generate the next visual observation based on a scene and a specific robot action. This helps in training robots to perform complex tasks by predicting what the world should look like after a movement occurs. These capabilities are also being applied to the medical field for surgical simulations.

For surgery, the guidance describes models that generate video based on robot movements. These models can also predict actions from video or allow developers to test a specific policy as a procedure unfolds. This provides a safe way to practice complex medical maneuvers in a virtual space before performing them on patients.

Hardware requirements and speed

NVIDIA provided a detailed generation table to help developers understand the GPU memory needs for different models. The Cosmos3-Nano model requires 58 GiB of GPU memory in BF16 precision. However, this can be reduced to 44 GiB in FP8 or as low as 31 GiB if using FP8 with offloading, which is a method of moving data between memory types.

The larger Cosmos3-Super model has much higher requirements, needing 150 GiB in BF16. It can run in FP8 with 93 GiB of memory or 38 GiB with offloading. These specifications help engineers determine which hardware they need to purchase or allocate to run these specific world models effectively.

To improve performance, NVIDIA offers Cosmos NIM packages, which are optimized containers for inference. These packages can use speculative decoding to achieve mean speedups of roughly two to three times on H200 GPUs. Additionally, streaming video-language-model inference can enable subsecond alerts with 10 times lower latency than standard methods.

Compatibility and next steps

The Cosmos framework is designed to work with several existing models, including Alpamayo, GR00T, Qwen, and pi0.5. By supporting these models alongside Cosmos 3, NVIDIA is creating a more interconnected ecosystem for developers. Certified Cosmos NIM supports deployment on more than 20 different NVIDIA GPUs.

These new recipes add significant implementation detail to the tools NVIDIA has already released. They bridge the gap between raw model capabilities and practical, real-world deployment in industrial settings. Developers can now see exactly how to map their data and hardware to the Cosmos framework.

NVIDIA has also scheduled a livestream for November 5 to provide further information. This upcoming event will likely offer more clarity on how these models perform in diverse environments. For now, the released recipes serve as the primary technical roadmap for the community.