SI Agent News
DevelopingClaudeOpenAIGeminiAgents

New benchmark tests if AI can code for robots in real-time

The Embodied First Exam evaluates whether autonomous coding agents can reliably manage robotic hardware across complex physical environments.

Isometric diorama: picture for the story 'New benchmark tests if AI can code for robots in real-time'
AI-made isometric diorama for this story. No real people or products are shown.
33.6%Success rate of top model in standard setting

The short version

Why it matters

This benchmark provides a standardized way to measure how close AI is to physical autonomy. It highlights that while AI can write code, it still lacks the 'physical common-sense' needed to handle real-world objects safely.

A new standard for physical AI

A consortium of researchers from institutions including Stanford, UC Berkeley, and Carnegie Mellon has introduced the Embodied First Exam. This open-access diagnostic evaluation benchmark is designed to see if autonomous coding agents can reliably direct robotic hardware in complex physical environments.

While previous benchmarks focused on linguistic reasoning or simple tabletop tasks, this new framework addresses a major gap in the field. It provides a unified way to test whether models can solve diverse physical problems while facing authentic physical constraints.

The benchmark is built on a modular, containerized architecture derived from Terminal-Bench. It includes nine new task suites that cover advanced capabilities like 3D spatial puzzle solving and multi-robot coordination.

Three ways to test the agents

The researchers established three distinct evaluation protocols to provide a reproducible way to test different types of AI capabilities. The Standard Setting emulates the real world by giving the agent only realistic sensor data, such as camera feeds, without any hidden information.

The Realtime Setting adds a layer of difficulty by keeping the physics simulator clock running while the agent thinks and compiles its code. This specific test measures whether an agent can react quickly enough to environmental momentum or if it is too slow to be useful in a moving world.

Finally, the Privileged Setting acts as an oracle by giving the agent unlimited resets and access to ground-truth simulator states. This configuration helps researchers identify the theoretical upper limits of what a model can do and helps generate expert demonstrations for future training.

Where current models fail

The study tested eight state-of-the-art coding agents on a core suite of 456 tasks. In the realistic Standard Setting, the top-performing system, GPT-6 Astra, achieved a task success rate of 33.6%. When pooling the top three models together, they still left more than half of the tasks unsolved.

Performance dropped sharply when moving to the Realtime Setting. In a matched comparison, 63% of the tasks remained completely unsolved by any model in real-time mode. The authors noted that long chain-of-thought reasoning windows often cause delays that make robots miss critical windows for manipulation.

The analysis also revealed specific bottlenecks where agents consistently fail. For example, tasks requiring the deformation of materials failed 66% of the time once reached. Dynamic whole-body balance also showed high failure rates, highlighting a lack of physical common-sense reasoning in current models.

Agents as meta-engineers

One interesting finding is how agents choose their own methods for solving problems. In standard settings, agents mostly used direct control by issuing simple commands step-by-step. However, in privileged and real-time settings, they overwhelmingly chose to synthesize complete control programs using motion planning libraries.

The benchmark also observed a shift toward agents acting as meta-engineers. In complex scenarios like humanoid soccer matches, agents with GPU resources autonomously chose to train deep reinforcement learning policies. These were the only attempts that successfully won matches, showing that AI can learn to tune lower-level policies on the fly.

This behavior suggests that the future of robotics may not just be about better code. It may be about agents that can identify which type of learning or programming is best suited for a specific physical goal and then execute that strategy.

Safety and infrastructure needs

The results of the Embodied First Exam have significant implications for how university labs and research offices manage robotics. Because leading agents fail over 66% of multi-step tasks, the researchers warn that autonomous code execution cannot be safely deployed to physical testbeds without human-in-the-loop interlocks.

The study also highlights a growing need for high-performance computing infrastructure. Because agents are beginning to perform on-the-fly reinforcement learning, research facilities must support multi-container virtualization and GPU-accelerated simulations alongside traditional workloads.

Finally, the benchmark sets a precedent for transparent reporting in the research community. It encourages the separation of 'privileged' oracle metrics from realistic performance to ensure that success rates are not inflated by giving AI information it wouldn't have in the real world.

What to watch

Try it

Look up the Embodied First Exam on arXiv to see the full list of 1,156 tasks and the specific failure taxonomy.

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by CASRAI · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news