SIAgentNews.com
← All news

A new benchmark asks whether agent skills help once the conditions stop being ideal

SkillBuffBench tests coding-agent skills in realistic conditions: a crowded library, optional memory, many models. No results yet, but the design is worth knowing.

· First reported by arxiv.org

fouragent harnesses, each run with every model tested

The short version

  • Skills are file-based packages of how-to knowledge that a coding agent can find and load when needed.
  • Earlier evidence mostly hand-picked the right skill and started the agent with no memory.
  • The test compares no skill, a curated skill, and automatic pick-up from an installed library, with memory off and on.
  • Every model runs under every one of four agent harnesses, on expert-written tasks in materials science and biology.
  • Results are not out yet; the paper says they will be added as runs finish.

Why it matters

Skills are now built into every major agent tool, and many people assume they make agents better. This benchmark is an attempt to check that under conditions closer to real use. Until results arrive, the useful lesson is to measure your own setup instead of trusting the assumption.

What the paper sets out to do

As arxiv.org reported in a new preprint (a research paper shared before peer review), a team has introduced SkillBuffBench. It is a benchmark, meaning a standard set of tasks used to compare systems, along with a protocol for running paired comparisons.

The target is a feature that has become common in AI coding tools: Agent Skills. These are file-based packages of procedural knowledge, basically written know-how, that an agent can discover and load when a task calls for it. The paper notes that every major agent harness now ships them. A harness is the software wrapper that runs a model and gives it tools to work with.

The authors' complaint is about the evidence. Claims that skills help mostly come from tests where someone picked the right skill for each task in advance, and where the agent began with an empty memory.

Why the old tests are too tidy

Real use looks different, the authors argue. An agent usually faces whatever library of skills happens to be installed, and it has to decide for itself which one, if any, fits the job. It also often carries memory from earlier work.

Think of a new hire handed a shelf of manuals. Being given the one correct manual is a very different situation from being left to find it among dozens, while also remembering last week's projects. A test that only covers the first situation may overstate how much the manuals help.

SkillBuffBench is built to cover the messier case, so that a score reflects what skills contribute under deployment-like conditions.

How the experiment is laid out

The design treats skill access as three levels: no skill at all, a curated skill supplied for the task, and automatic invocation, where the agent picks from an installed library. Each of these is crossed with memory switched off and on.

That grid is repeated across four agent harnesses and several models. The models include leading proprietary ones from Anthropic, Google and OpenAI, plus leading open-weight ones from Chinese providers. Open-weight means the trained model files are published so others can run them. Every model is run under every harness, so differences can be traced to the model, the harness, the skill setting or memory.

The tasks are research problems contributed by domain experts, who also supplied the skills used. For now they cover materials science and biology, and the suite is meant to grow on a rolling basis.

How results will be judged

Each run is graded by a deterministic verifier, which means an automatic checker that gives the same verdict every time instead of a human or another AI forming an opinion. Uncertainty is reported using a paired, task-level bootstrap. In plain terms, that is a statistical method that resamples the results to show how much a difference could be down to chance, with each task compared against itself under different settings.

The paper also reports cost, the number of agent rounds and the wall-clock time next to the score, rather than blending them into a single number. That matters because a skill that lifts accuracy slightly but doubles the bill is a different proposition from one that is cheap and helpful.

What is not here yet

This version of the paper describes the benchmark design, the protocol, and the current tasks and skills. The evaluation is still in progress. The authors say results will be added when the runs are done, and that the tasks, results and analysis will be updated over time.

So there is no finding to quote, no ranking of models and no verdict on whether skills help. Anyone repeating a number from this paper right now would be inventing it. What exists is a plan, and a fairly careful one.

The audience is also narrow: people who build or choose agent tools. For them, the value is the framing, which separates the benefit of a skill from the benefit of picking the right one, and from the effect of memory.

What to watch

  • Whether the first results show skills helping when the agent must pick from an installed library, not just when handed the right one.
  • Whether memory changes how much skills contribute.
  • Whether the paper's cost, rounds and time figures show any skill benefit comes at a price.

Try it

Pick one task you give a coding agent and run it twice on the same model, once with your skill file loaded and once without. Compare quality, cost and time, and keep the skill only if it clearly helps.

Sources and links

First reported by arxiv.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news