The agent wrapper around a coding model can swing results by 44 points

A test built around pairings

Some software teams cannot send their code to an outside AI service. Their proprietary code, and the records of how an agent worked on it, have to stay on infrastructure they control. For them, small open-weight models (models whose files anyone can download and run) are attractive because they need less hardware.

But a model alone does not fix bugs in a codebase. It has to be placed inside a harness, the surrounding software that feeds it tasks and tools and lets it act on a project. As arxiv.org shows in this new preprint (a paper not yet peer reviewed), most public leaderboards report one fixed model-and-harness combination, so readers cannot tell which part deserves the credit.

The authors set out to separate the two. They ran a full-factorial test, meaning every harness was paired with every model. That covered six open-source harnesses and three lightweight coding models, each attempting 100 tasks from SWE-bench Verified, a benchmark of software-engineering problems. The result was 1,800 graded runs, all served, metered and graded the same way.

Same model, very different results

The headline finding is how much the harness matters. Holding the model constant, the share of tasks resolved differed by as much as 44 percentage points depending on which harness was used. That is not a small tuning effect. It is the difference between a tool that mostly works and one that mostly does not.

The rankings also did not hold still. The harness that looked best with one model was not reliably best with another. So you cannot pick a winning harness once and assume it will carry over if you later swap the model underneath it.

The authors draw a blunt conclusion: strong performance with one harness or one model does not guarantee similar results with a different one. In their view, choosing the pairing becomes a central deployment decision rather than a footnote.

How the failures differ

The team did not stop at scores. They applied an evidence-based failure taxonomy, a structured way of sorting what went wrong, to 1,100 of the unsuccessful runs. The goal was to see not just how often agents failed but how.

Failure patterns varied a lot across pairings. A model's most common way of failing under one harness did not necessarily remain its most common way under another. The same model can stumble for different reasons depending on the setup around it.

This has a practical consequence. If you only track a pass rate, you may fix the wrong thing. Knowing whether a pairing tends to fail in one way or another tells you where a person should check the work, and where adjusting the harness could help.

The cost side: tokens per solved task

Efficiency also depended on the pairing. The paper measures it in charged tokens per resolved task. Tokens are the chunks of text a model reads and writes, and they are the usual unit for metering usage. Dividing by resolved tasks matters because it rewards getting the job done, not merely producing lots of activity.

Even among harnesses where the model itself directs the loop, meaning it decides its next step repeatedly, that figure varied by more than 20-fold. Two setups that look alike on paper can differ enormously in what each fixed bug costs in compute.

For self-hosted teams, this is a hardware and scheduling question as much as a billing one. A pairing that burns many times more tokens needs more capacity to deliver the same number of fixes.

What teams can do with it

The authors turn their results into guidance on three fronts: how to select a pairing, how to assess token efficiency, and how to use run records to improve a harness. They also released their complete experimental records and analysis files, so others can inspect the data or compare their own setups.

The caveats are worth keeping in view. The study used 100 tasks from one benchmark and three lightweight models, so it shows that pairing effects can be large, not that any particular combination is best for your codebase. The most reliable lesson is about method: test combinations together, record failures, and measure cost per solved task.