SI Agent News
DevelopingClaudeOpenAIGeminiAgents

WebArena-Pro tests web agents on 20 self-hosted apps, with audio and video tasks

A new arXiv benchmark rates AI agents on 300 browser tasks, including audio and video, and the best scores are 62.3%, so web agents still fall short.

62.3%best score on WebArena-Pro, tied by two agent setups

The short version

Why it matters

Vendors often say their agents can handle web chores, but the tests behind those claims are usually narrow or impossible to repeat. A benchmark that others can rerun, with varied apps and media, gives a firmer yardstick. Even the best score here leaves a large share of tasks unsolved, which is a useful check on hype.

What the paper proposes

As arxiv.org reported, researchers have released WebArena-Pro, a test for AI agents that do work in a web browser. It holds 300 tasks spread over 20 web applications, grouped into six domain categories.

The applications differ in how they look, how their workflows run and how their data is organized. That variety is the point: real browser work rarely stays inside one site.

The gap it tries to fill

The authors argue that existing reproducible benchmarks, meaning ones that give comparable results every time, cover few apps from a few software types. They also limit inputs to text and images.

Tests run on the live web cover more sites, but pages and data change between runs, so scores cannot be fairly compared. WebArena-Pro hosts its own applications to avoid that drift, and runs each task in isolation so many can be tested in parallel.

Audio and video count

The paper says current benchmarks do not really check whether an agent can understand audio and video inside a web task. Here those are core inputs, and the agent has dedicated actions to pull information from each.

Tasks are written through an annotator tool, screened with help from a language model, and finally validated by people before release.

How the agents scored

Two setups tied for first at 62.3%: Codex paired with a GPT model, and Claude Code paired with a Claude model. An OpenCode setup using Kimi followed at 60.7%.

The best results from the more general-purpose agent setups, using a Gemini model and a Gemma model, were 30.3% each. In short, even the leaders miss well over a third of the tasks.

What to watch

Try it

When a vendor claims its agent can handle web tasks, ask which benchmark backs it and whether others can rerun it. Then test the agent on one real task of yours.

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by arxiv.org · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news