NVIDIA launches GR00T 2 to train robots using human video data

Learning from human movement

NVIDIA introduced GR00T 2 at the Conference on Robot Learning in Austin. The model is designed to help robots learn complex actions by observing how humans perform them in the real world. This approach allows robots to gain broader capabilities without needing constant, direct human intervention during every training session.

To achieve this, NVIDIA pretrained the model on a massive dataset. This includes over 402,730 hours of first-person human video and 23,500 hours of robot teleoperation. By feeding the model these diverse examples, the researchers aim to create a foundation where robots can understand the intent behind human movements.

The results show that this training method works effectively for new tasks. As reported by Unite.AI, the model achieved a 40% zero-shot success rate on 50 tasks that were completely absent from its training data. This means the robot could perform actions it had never specifically seen before by relying on its general understanding of human behavior.

How the model processes data

GR00T 2 is built on a 2-billion-parameter transformer based on Cosmos-Predict2.5-2B. It functions by combining several different inputs to understand a situation. These inputs include recent camera views, the current state of the robot, and a written instruction provided by a human user.

Once the model receives these inputs, it generates two things: the expected future views and the next sequence of actions. This allows the robot to predict what should happen next based on its training. It combines these elements to handle complex movements that require precise coordination.

The model is versatile enough to cover different types of robotic bodies. A single saved version of the model supports various forms, including humanoid shapes, hands with many fingers, and arms equipped with parallel grippers. This allows for cross-embodiment tests, which show how skills can transfer between different robot bodies and adapt with only limited amounts of specific robot data.

Precision in simulation and assembly

The foundation for this training is Isaac Lab 3.0, which provides a unified API for physics, rendering, and visualization. NVIDIA reports that using the Newton backend for rigid bodies and particles can provide up to 2.5 times the training throughput of previous versions. This allows for more efficient large-scale practice in virtual environments.

One specific application is AlphaNIST, which uses large-scale simulated practice for 20 NIST Assembly Task Board tasks. These tasks involve parts as small as 4 millimeters. A shared policy trained only with a yes-or-no success reward achieved a 98.2% training success rate, using roughly 39 times less GPU time than separate per-task policies.

This success is driven by Success-Guided Sampling, a method developed by the University of Washington and NVIDIA. This technique directs practice toward configurations near the policy's capability frontier, specifically those it solves about half the time. The authors report that this method scales effectively beyond one million simulated environments and can transfer to real hardware.

Real-world testing and industrial use

NVIDIA has demonstrated the model's ability to handle complex assembly from randomly dumped parts. With a reasoning agent to sequence skills and monitor progress, the model successfully built 98 toy planes, 97 toy tractors, and 97 Hot Wheels cars out of 100 trials each. In contrast, supervised fine-tuning alone produced only 69, 51, and 23 successful assemblies, respectively.

The model also supports a variety of industrial benchmarks and libraries. The VABAR project from UC Berkeley and NVIDIA covers tasks like tool sorting, gear meshing, and cable routing, supported by a 33-skill library. Additionally, the Isaac Lab-Arena framework provides a way to run repeatable evaluations across GPU-accelerated simulated environments to analyze failures.

Several companies are already testing the technology for practical applications. Nexuni is testing GR00T 2 for laundry handling, while Foxconn is applying it to server-manufacturing automation. Siemens is also currently evaluating the model for various industrial use cases. While these tests are ongoing, no official production rollout has been announced yet.