Claude Fable 5.2 leads Google's visual office work benchmark

A new test for visual AI

Google DeepMind and Artificial Analysis recently launched the Multimodal Knowledge Agent (MKA) benchmark. This test was created to see if AI agents can handle complex office work that requires looking at visual files like blueprints, radiographs, and maps. Most existing benchmarks only use text, but many professional jobs rely heavily on visual data.

To build the test, the researchers worked with domain experts across 10 different sectors, including construction, healthcare, and law. These specialists wrote 200 tasks that represent multi-step projects. Each task is designed to take a human professional between three and eight hours to complete from start to finish.

The benchmark evaluates how well an agent can turn these visual inputs into finished products like slide decks, spreadsheets, or PDF reports. Each task includes a rubric with roughly 70 checks. These checks are weighted so that 75% of the score comes from factual accuracy and 25% comes from the quality of the visual design.

How the models were graded

The MKA benchmark uses a specific environment called Stirrup to run the models. In this sandbox, agents are given a prompt and a bundle of files, such as CAD drawings or CT scans. The agents are allowed to use tools like code execution, web search, and an image-viewing tool to complete their work without human help.

Because the agents cannot 'see' a PDF or a slide directly, they must use code to render the file into an image first. This forces the model to interact with the data in a way that mimics how a human might open and analyze a document. The agents are given a context window of up to 1 million tokens and an 8-hour time limit to finish each project.

To ensure the grading was fair, Google used an 'agentic autorater' built on a non-Google model. This grader opens every file and inspects formulas and layouts against the expert rubric. When human experts graded the same outputs, they agreed with this automated grader 96% of the time.

Accuracy vs. cost efficiency

The results show that Anthropic's Claude Fable 5.2 reached the highest overall weighted satisfaction rate of 73.26%. However, Google's researchers noted that the gap between the top three models is statistically indistinguishable. This means that while Claude Fable 5.2 won, the difference in quality compared to the second-place model is very small.

The cost difference between the models is much larger. Gemini 4 Carbon spent an average of $5.63 per task at list prices, while Claude Fable 5.2 spent $35.28. This means the top-scoring model was about six times more expensive to run than the second-place model for nearly the same level of performance.

The report also looked at 'climb' checks, which are tasks that test new frontier capabilities. On these specific checks, Claude Fable 5.2 led Gemini 4 Carbon by 2.22 percentage points. Despite this lead, the report suggests that Carbon sits on the 'pareto frontier,' representing a highly efficient balance of quality and price.

What the data reveals

The benchmark highlights how much effort different models put into a single task. The models tested used between 30 and 130 turns on average to finish a project. Claude Fable 5.2 used 5.74 times as many total tokens as Gemini 4 Carbon and took nearly three times as many turns to reach its conclusion.

The study also explored what happens when an agent loses its ability to see images. When the image-viewing tool was removed from Gemini 4 Carbon on certain tests, its performance barely changed. The researchers attribute this to the fact that many tasks in those specific tests relied on spreadsheets rather than visual layouts.

For users, these findings suggest that the 'best' model isn't always the best choice for production. While Claude Fable 5.2 shows high-end capabilities, Gemini 4 Carbon offers a way to perform similar visual analysis at a fraction of the price. This distinction will be important as companies look to deploy agents for high-volume office work.