Berkeley and NVIDIA launch framework to turn robot videos into code
A new agentic robotics framework allows researchers to generate validated robot code from a single video demonstration using automated simulations.
The short version
- The system uses AI agents to write, test, and debug modular robot code in simulation environments.
- Researchers can turn a single video demonstration into validated code without needing large fleets of physical robots.
- The framework reached a 95.0% success rate on multi-tool tasks after 36 automated revisions.
- The final output is human-readable code that runs without a generative model in the real-time control loop.
Why it matters
This shift allows smaller research labs to compete with large corporations by removing the need for expensive physical hardware and human operators. It also improves safety by allowing researchers to inspect and test robot logic in a digital twin before deploying it to a real machine.
Bridging the robotics data gap
Robotics research has historically faced a difficult choice between two methods. Engineers could manually script every movement, which is time-consuming, or they could use machine learning to let robots learn from human demonstrations. However, there is a severe shortage of high-quality physical data compared to the trillions of text tokens available for language models.
UC Berkeley and NVIDIA are addressing this bottleneck with a new agentic robotics framework. Instead of requiring thousands of hours of human-led physical manipulation, the system uses foundation models to coordinate a team of specialized AI agents. These agents handle the heavy lifting of writing, inspecting, and refining the robot's control logic.
This approach moves the trial-and-error phase of learning into a physics engine. By performing these tests in a virtual environment, researchers can train complex behaviors without the risk of breaking expensive hardware. As reported by CASRAI, this shift allows for more efficient development of robot policies.
From video to digital twins
The process begins with a single real-world video of a task. The system uses multimodal foundation models and a tool called RoboSimStudio to reconstruct a digital twin. This digital twin mimics the real-world geometry and material physics, such as how a sponge might deform, in less than an hour.
Once the simulation is ready, coding agents structure the task using a Graph-as-Policy architecture. Rather than using a black-box neural network to move the robot, the agents create interpretable programs. These programs combine specific capabilities like perception, grasping, and trajectory planning into a logical sequence.
This modular approach ensures that the robot's actions are predictable. The agents can then run hundreds of parallel simulations with different lighting, object positions, and surface frictions. If a simulated arm fails a task, the agents analyze the failure video and update the physical model or the code script automatically.
Deterministic real-world control
The final stage of the pipeline involves exporting the verified code into Robot Operating System (ROS2) nodes. These nodes are then deployed to physical manipulators. A key technical detail is that the system does not run a generative foundation model during the robot's real-time movements.
By keeping the real-time control loop deterministic, the system ensures consistent timing and safety. The robot follows the specific, verified instructions generated during the simulation phase rather than
A verifiable three-stage pipeline
The autonomous control pipeline developed at UC Berkeley’s AUTOLab operates across three discrete and verifiable stages. The process begins with Real2Sim, where a video demonstration is converted into a digital environment. This is followed by code generation with recursive self-improvement, where agents refine the robot's logic through repeated testing. The final stage is Sim2Real transfer, which moves the validated instructions to the physical hardware.
This structured workflow ensures that every step of the robot's learning process is traceable. By separating the simulation, the logic generation, and the final deployment, researchers can pinpoint exactly where a failure occurs. This methodology allows for a systematic transition from a simple visual observation to a functional, high-performing robot operation.
Aligning with research priorities
This architectural shift aligns directly with institutional priorities regarding rigorous code verification and reproducible computational science. Because the pipeline generates human-readable code graphs instead of opaque weights, researchers can inspect and unit-test individual modules before they ever touch a physical machine. This transparency makes it easier for safety compliance officers to review the robot's intended actions.
Furthermore, using standardized simulation environments promotes reproducible AI experiments. It allows research teams to evaluate new algorithms against synthetic benchmarks without the risk of hardware degradation. By adopting open modular architectures, the framework supports the FAIR principles for research software, ensuring that academic algorithms can be shared and verified across independent laboratories worldwide.
What to watch
- How quickly the system can adapt to materials with highly complex physics that are difficult to simulate.
- Whether this framework will become the standard for university robotics labs to compete with corporate research.
Try it
Look up the AUTOLab project repository to see the open-source documentation for the agentic robotics framework.
Sources and links
- Paperarxiv.org/abs/2610.18835 arxiv.org
- ReportingCASRAI: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by CASRAI · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
