Bigger AI models did not handle broken tools any better in a new test
A preprint injected realistic tool failures into AI agents and found a larger model was no sturdier, so error handling around the model may matter more than size.
The short version
- MCP-Injector is a new tool that simulates failed or late responses on the path between an agent and its tools.
- Fault types were modeled on 57,240 issue and pull-request threads from 1,774 open-source MCP repositories.
- Under faults, average task-completion score fell by 60%, from 8.08 to 3.23 out of 10.
- The 120B model's completion rate under faults was 91.1%, versus 95.6% for the 20B model.
- Agent stacks differed in how well they absorbed faults, pointing to framework-level error handling.
Why it matters
If you build or buy AI agents that call tools, paying for a bigger model may not protect you when a tool breaks. How the surrounding software handles errors may count for more. Benchmarks that assume tools always work can hide this weakness.
Testing what happens when tools break
Many AI agents now use outside tools through the Model Context Protocol (MCP), a standard way for an agent to call services such as search or file access. Most tests of agents assume those tools answer correctly and on time. A new preprint, as arxiv.org listed it, asks what happens when they do not.
The authors built MCP-Injector, software that sits between the agent and its tools and deliberately causes failures. Faults can be permanent, transient (they go away), or intermittent (they come and go), and the schedules can be repeated exactly.
Where the failures came from
The researchers did not guess at what goes wrong. They based the mix of failures on the public record: 57,240 issue and pull-request threads across 1,774 open-source MCP projects, where developers report problems.
That mix leans toward permanent failures, so it is a harsh test. The authors call it a stress profile, and it should be read that way.
What the numbers show
The experiment had 1,350 cells, combining two model sizes (20B and 120B), three agent stacks (a custom plan-and-execute setup, the OpenAI Agents SDK, and LangChain), and three fault conditions. Under the injected faults, the average task-completion score dropped from 8.08 to 3.23 out of 10.
The larger model did not hold up better. The 120B version kept no bigger share of its fault-free score (p = 0.96), and its completion rate was 91.1% against 95.6% for the smaller one. That gap has a confidence interval running from -0.2 to 9.3 points, so it is suggestive, not proof of a real difference.
Limits and the practical lesson
Only two models were tested, both from the gpt-oss family, so the finding may not hold for other models. This is also a preprint, meaning it has not been peer reviewed.
Still, the authors saw clear differences between agent stacks. They read that as a sign that framework-level error handling, such as retries and checks on tool output, matters more than model size.
What to watch
- Will the result hold for other model families and sizes beyond the two gpt-oss models tested?
- Will agent frameworks add built-in fault handling, and will other benchmarks start injecting tool failures?
Try it
Pick one workflow where an agent calls a tool, deliberately make that tool time out or return junk, and watch whether the agent retries, reports the failure, or carries on as if it worked.
Sources and links
- Reportingarxiv.org: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by arxiv.org · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.