WebArena-Pro tests web agents on 20 self-hosted apps, with audio and video tasks
A new arXiv benchmark rates AI agents on 300 browser tasks, including audio and video, and the best scores are 62.3%, so web agents still fall short.
The short version
- 300 tasks across 20 self-hosted web applications in six domain categories
- Audio and video are treated as core inputs, alongside text and images
- Two agent setups tie for the top score at 62.3%; a third reaches 60.7%
- The strongest GenericAgent setups both score 30.3%
- Tasks are written by annotators, screened with LLM help, then checked by humans
Why it matters
Vendors often say their agents can handle web chores, but the tests behind those claims are usually narrow or impossible to repeat. A benchmark that others can rerun, with varied apps and media, gives a firmer yardstick. Even the best score here leaves a large share of tasks unsolved, which is a useful check on hype.
What the paper proposes
As arxiv.org reported, researchers have released WebArena-Pro, a test for AI agents that do work in a web browser. It holds 300 tasks spread over 20 web applications, grouped into six domain categories.
The applications differ in how they look, how their workflows run and how their data is organized. That variety is the point: real browser work rarely stays inside one site.
The gap it tries to fill
The authors argue that existing reproducible benchmarks, meaning ones that give comparable results every time, cover few apps from a few software types. They also limit inputs to text and images.
Tests run on the live web cover more sites, but pages and data change between runs, so scores cannot be fairly compared. WebArena-Pro hosts its own applications to avoid that drift, and runs each task in isolation so many can be tested in parallel.
Audio and video count
The paper says current benchmarks do not really check whether an agent can understand audio and video inside a web task. Here those are core inputs, and the agent has dedicated actions to pull information from each.
Tasks are written through an annotator tool, screened with help from a language model, and finally validated by people before release.
How the agents scored
Two setups tied for first at 62.3%: Codex paired with a GPT model, and Claude Code paired with a Claude model. An OpenCode setup using Kimi followed at 60.7%.
The best results from the more general-purpose agent setups, using a Gemini model and a Gemma model, were 30.3% each. In short, even the leaders miss well over a third of the tasks.
What to watch
- Will independent teams rerun WebArena-Pro and get similar scores?
- Does the preprint pass peer review, and do the task set and scoring hold up?
- How quickly do agents close the gap on audio and video tasks?
Try it
When a vendor claims its agent can handle web tasks, ask which benchmark backs it and whether others can rerun it. Then test the agent on one real task of yours.
Sources and links
- Reportingarxiv.org: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by arxiv.org · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
