A new test measures how AI agents perform as teams, not just alone
MultiAgentBench scores AI agents on working together and competing, and finds team structure and planning affect results.
The short version
- The benchmark tests LLM agents in interactive scenarios that involve both collaboration and competition.
- It uses milestone-based indicators to score progress, not only whether the final task was completed.
- It compares four coordination layouts: star, chain, tree and graph.
- Cognitive planning raised milestone achievement rates by 3%, according to the abstract.
- Code and datasets are public, so others can rerun and extend the tests.
Why it matters
More AI products now chain several agents together, yet most tests only check one agent alone. A shared yardstick for groups helps builders and buyers compare setups on more than a final answer. Treat the scores as a guide, since a benchmark result does not prove a system will work in your own workplace.
A test built for teams of agents
Researchers have posted a paper called MultiAgentBench, which sets out to measure how AI agents behave when they work with, or against, each other. As arxiv.org shows, it was first submitted on 3 Mar 2025 and revised in a second version dated 18 Nov 2026. This article draws on the abstract of that revised version.
The authors start from a gap. Large language models (LLMs, the text-based AI systems behind chatbots) now act as autonomous agents, meaning they take steps toward a goal on their own. But the authors say existing benchmarks, which are standardized tests, either look at one agent working alone or stay within narrow domains. Neither captures what happens when several agents must coordinate or compete.
Scoring progress, not only the finish line
The benchmark covers a range of interactive scenarios. It measures whether a task got done, and it also tries to rate the quality of the collaboration and the competition. To do that, the authors introduce what they call milestone-based key performance indicators.
In plain terms, that means checking progress against set milestones along the way, not only the end result. This matters because a group can reach the right answer by luck or by good teamwork, and a final-answer score alone cannot tell the two apart. The abstract does not spell out how the milestones are defined, so the full paper is the place to check.
Different team shapes, different results
The researchers also compared coordination protocols, which are the patterns that decide who talks to whom. They tested four: star, chain, tree and graph. Roughly, a star routes everything through a central hub, a chain passes work from one agent to the next, a tree branches out from a lead, and a graph allows more open links between agents.
They also tried two strategies they describe as innovative: group discussion and cognitive planning. The abstract does not explain the mechanics of either, so we will not guess at them.
Three results are highlighted:
- gpt-4o-mini reached the highest average task score.
- The graph structure performed best among the coordination protocols in the research scenario.
- Cognitive planning improved milestone achievement rates by 3%.
These come from the abstract only. The graph result applies to the research scenario specifically, and the abstract does not say it held everywhere. A 3% gain is also a modest figure, and we have not seen how it was measured.
What this changes for builders and buyers
Teams of agents fail in ways that a single agent does not. One can be strong alone and still duplicate a teammate's work or follow a poor instruction from another agent. A benchmark aimed at groups gives developers a common way to compare designs, and it gives technical buyers something to ask vendors about.
There is a limit, though. Benchmark scenarios are controlled, and a high score in one does not guarantee the same result on your company's tasks. The sensible use is as a starting point for questions, such as which team layout suits a job and whether progress was tracked or only the final output.
The paper also says its code and datasets are public. That lets other researchers repeat the tests or add their own scenarios, which is how a benchmark becomes a shared standard rather than one lab's claim.
What we do not know yet
The abstract leaves several questions open. It does not list every model tested beyond naming the top scorer, and it gives no detail on how each scenario is built. It also does not say how the milestone indicators were checked for fairness.
Because this is a revised paper, the numbers may differ from the first posting, and the abstract is only a summary. Anyone planning to rely on the findings should read the full paper and look at the released code before drawing conclusions.
What to watch
- Do other researchers reuse the benchmark and report results that match the graph and cognitive planning findings?
- How exactly are the milestone indicators defined, and do they separate good teamwork from lucky outcomes?
- Do the results hold for models other than the ones tested and for scenarios beyond research?
Try it
If you build or buy multi-agent tools, open the paper's public code and datasets and check whether any scenario resembles your use case.
Sources and links
- Reportingarxiv.org: the article we sourced
First reported by arxiv.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.