New MKA benchmark tests whether AI agents can do professional work that needs reading images
A new arXiv paper tests AI agents on 200 professional tasks built around blueprints, scans and maps, and even the best models leave a lot of room to improve.
The short version
- MKA has 200 tasks spread across 10 economic sectors.
- Tasks use visually dense files such as CAD blueprints, schematics, radiographs, charts and maps.
- Agents must produce a finished file, such as .xlsx, .pptx, .pdf, .docx, .html or .png.
- Claude Fable 5.2 scored 73.3% overall WSR; Gemini 4 Carbon scored 71.9%.
- Gemini 4 Carbon used 5.7× fewer tokens than the top scorer.
Why it matters
Much real office and engineering work depends on reading drawings, scans and charts, not just text. A test built around that shows how far agents are from handling it alone. The paper is a preprint, so the numbers are the authors' own and have not been independently checked.
What the benchmark asks agents to do
As arxiv.org reported, researchers have introduced the Multimodal Knowledge Agent benchmark, or MKA. It measures how well autonomous agents (AI programs that carry out multi-step work on their own) handle professional tasks that depend on understanding images.
Each task hands the agent several visually heavy files. Examples include CAD blueprints, electronics schematics, radiographs (X-ray images), complex charts and geospatial maps. The agent has to combine what it finds and deliver a finished document in a format like a spreadsheet, slide deck, PDF, Word file, web page or image.
Why the authors built a new test
The authors argue that existing benchmarks barely test this skill. They point to one popular benchmark, GDPval, where taking away an agent's image-viewing tools did not change the score for Gemini 4 Carbon. That suggests the test never really needed the images.
They also say tests limited to a single file type do not force an agent to reason across several files or kinds of content. MKA is designed to close that gap, with 200 tasks across 10 economic sectors.
How the grading works
Deliverables are scored against rubrics weighted by experts. The grader is itself an AI agent running in a sandbox, a sealed environment. The authors say it agreed with human domain experts on 96.2% of individual criteria in their study.
Scores are reported as weighted satisfaction rate (WSR). One version covers everything, and a second, called Climb-WSR, covers only the harder criteria that earlier-generation models had not yet mastered.
What the results show
Four frontier models were tested. Claude Fable 5.2 led with 73.3% overall and 56.6% on Climb-WSR. Gemini 4 Carbon followed closely with 71.9% and 54.3%, while using 5.7× fewer tokens, the small units of text that models are billed on.
The other two models tested were Claude Opus 5.5 and GPT-6 Astra. Even the strongest models stayed under 57% on the hard criteria, which the authors read as plenty of headroom.
What to watch
- Whether model makers start publishing their own MKA scores.
- Whether other researchers reproduce the results with the AI grader.
Try it
When a vendor claims its agent can handle documents, ask whether the test required reading images and combining several files, or just text.
Sources and links
- Reportingarxiv.org: the article we sourced
First reported by arxiv.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.