A new test lab checks how AI assistants cope with simulated people doing real tasks

What the researchers built

As arxiv.org reported, a research team has released Interactive Visual Gym, a framework for testing AI models that look at a scene and guide a person through a physical or on-screen task. Think of an assistant that watches you cook and tells you what to do next.

The key idea is a closed loop. In a closed loop, the assistant's reply changes what the user does, which changes what the assistant sees next. The authors say existing benchmarks mostly replay recorded sessions, so they cannot capture this back-and-forth.

Simulated people, simulated kitchens

The team made three simulators that recreate everyday tasks: cooking, household manipulation and mobile-interface use. A simulated user, built to match a given persona, works with the assistant inside these worlds.

That simulated user does not follow a perfect script. It can make controlled execution errors and do things outside the plan, much as real people do. The authors packaged the setup as IVG-Bench, a benchmark of 315 interaction scenarios.

How current models performed

The authors tested leading vision-language models, meaning AI systems that handle both images and text. They report substantial limitations. The best performer, Gemini 4 Argon, completed only 53.4% of tasks within the time limit.

The paper also points to weak factual grounding (staying accurate about what is actually in view) and weak actionable guidance. Models were often too slow as well, so a warning could arrive after the mistake had already happened.

Checks and early training results

To check that their scoring makes sense, the researchers ran a human study. Their interaction-quality metric agreed with participant preferences 78.5% of the time.

They also ran a proof-of-concept training experiment using GRPO, a reinforcement-learning method where a model learns from the outcomes it produces. In the cooking environment, closed-loop training improved both task success and interaction quality for Qwen3.5-4B. It is an early result from one environment, and the paper is a preprint that has not necessarily been peer reviewed.