New benchmark tests if AI can code for robots in real-time
The Embodied First Exam evaluates whether autonomous coding agents can reliably manage robotic hardware across complex physical environments.

The short version
- The benchmark includes 1,156 tasks across 35 suites and 47 different robot bodies.
- Top models like GPT-6 Astra and Claude Opus 5.5 achieved only a 33.6% success rate in realistic settings.
- Agents showed a major performance drop in real-time modes due to reasoning latency.
- The framework allows agents to choose between direct commands, code synthesis, or training their own policies.
Why it matters
This benchmark provides a standardized way to measure how close AI is to physical autonomy. It highlights that while AI can write code, it still lacks the 'physical common-sense' needed to handle real-world objects safely.
A new standard for physical AI
A consortium of researchers from institutions including Stanford, UC Berkeley, and Carnegie Mellon has introduced the Embodied First Exam. This open-access diagnostic evaluation benchmark is designed to see if autonomous coding agents can reliably direct robotic hardware in complex physical environments.
While previous benchmarks focused on linguistic reasoning or simple tabletop tasks, this new framework addresses a major gap in the field. It provides a unified way to test whether models can solve diverse physical problems while facing authentic physical constraints.
The benchmark is built on a modular, containerized architecture derived from Terminal-Bench. It includes nine new task suites that cover advanced capabilities like 3D spatial puzzle solving and multi-robot coordination.
Three ways to test the agents
The researchers established three distinct evaluation protocols to provide a reproducible way to test different types of AI capabilities. The Standard Setting emulates the real world by giving the agent only realistic sensor data, such as camera feeds, without any hidden information.
The Realtime Setting adds a layer of difficulty by keeping the physics simulator clock running while the agent thinks and compiles its code. This specific test measures whether an agent can react quickly enough to environmental momentum or if it is too slow to be useful in a moving world.
Finally, the Privileged Setting acts as an oracle by giving the agent unlimited resets and access to ground-truth simulator states. This configuration helps researchers identify the theoretical upper limits of what a model can do and helps generate expert demonstrations for future training.
Where current models fail
The study tested eight state-of-the-art coding agents on a core suite of 456 tasks. In the realistic Standard Setting, the top-performing system, GPT-6 Astra, achieved a task success rate of 33.6%. When pooling the top three models together, they still left more than half of the tasks unsolved.
Performance dropped sharply when moving to the Realtime Setting. In a matched comparison, 63% of the tasks remained completely unsolved by any model in real-time mode. The authors noted that long chain-of-thought reasoning windows often cause delays that make robots miss critical windows for manipulation.
The analysis also revealed specific bottlenecks where agents consistently fail. For example, tasks requiring the deformation of materials failed 66% of the time once reached. Dynamic whole-body balance also showed high failure rates, highlighting a lack of physical common-sense reasoning in current models.
Agents as meta-engineers
One interesting finding is how agents choose their own methods for solving problems. In standard settings, agents mostly used direct control by issuing simple commands step-by-step. However, in privileged and real-time settings, they overwhelmingly chose to synthesize complete control programs using motion planning libraries.
The benchmark also observed a shift toward agents acting as meta-engineers. In complex scenarios like humanoid soccer matches, agents with GPU resources autonomously chose to train deep reinforcement learning policies. These were the only attempts that successfully won matches, showing that AI can learn to tune lower-level policies on the fly.
This behavior suggests that the future of robotics may not just be about better code. It may be about agents that can identify which type of learning or programming is best suited for a specific physical goal and then execute that strategy.
Safety and infrastructure needs
The results of the Embodied First Exam have significant implications for how university labs and research offices manage robotics. Because leading agents fail over 66% of multi-step tasks, the researchers warn that autonomous code execution cannot be safely deployed to physical testbeds without human-in-the-loop interlocks.
The study also highlights a growing need for high-performance computing infrastructure. Because agents are beginning to perform on-the-fly reinforcement learning, research facilities must support multi-container virtualization and GPU-accelerated simulations alongside traditional workloads.
Finally, the benchmark sets a precedent for transparent reporting in the research community. It encourages the separation of 'privileged' oracle metrics from realistic performance to ensure that success rates are not inflated by giving AI information it wouldn't have in the real world.
What to watch
- How quickly reasoning latency improves as models get faster.
- Whether agents can successfully navigate 'deformation' tasks without human intervention.
- The adoption of the benchmark by industrial robotics companies for safety testing.
Try it
Look up the Embodied First Exam on arXiv to see the full list of 1,156 tasks and the specific failure taxonomy.
Sources and links
- ReportingCASRAI: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by CASRAI · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
