A new test changes the task mid-job, and every coding agent did worse

A test where the ground moves

Most tests of AI coding agents hand the agent a task and a fixed set of files, then check whether it finishes. As arxiv.org shows in a new paper, that setup assumes nothing changes while the agent works. Real development rarely behaves that way. Code gets edited, software dependencies (the outside libraries a project relies on) get updated, the state of the running system shifts, and requirements get revised.

The paper introduces SWE-Shift, a benchmark built around those mid-task changes. It was submitted on 23 Oct 2026. The aim is to measure how an agent behaves when the job it was given is no longer quite the job in front of it.

How the benchmark is built

The design is a paired comparison. Researchers take one saved history and workspace and run it two ways. In the Clean continuation, the agent carries on undisturbed. In the matching Shifted continuation, it receives an update relevant to the task.

Because both runs start from the same point, any gap between them comes from the change rather than from a different starting situation. The benchmark holds 120 scenarios across 12 change families, and each is tested at nine scheduled points where the update can arrive.

Passing is also stricter than "the code works." An agent has to complete the current task, keep any earlier work that is still valid, and satisfy checks on the workspace state and on its evidence.

What the scores show

The authors tested 17 coding-agent systems. Every one did worse on the Shifted runs than on the Clean ones. There was no exception.

The paper gives one example. GPT-6 Astra succeeded 67.35% of the time on Clean runs but only 24.79% on Shifted runs. Its paired success was 20.34%. The abstract does not define that measure, but the figure is lower than the Shifted score alone.

Two kinds of change were especially hard. Changes to tools or sessions, and cases involving stale evidence, produced zero paired success across all systems. Not one of the 17 handled them under that measure.

Timing gave a mixed picture. Late-position paired success was lower for some systems and higher for others, so a late interruption is not always the worst case.

Finishing is not the same as recovering

The authors draw a pointed conclusion. An agent can succeed at the unchanged task and still fail to preserve work, recover state, or refresh evidence after an update.

Think of a contractor who finishes the room you first described. Halfway through you say the wall should move. A good contractor keeps the work that still stands, re-measures, and rechecks the plan. A poor one finishes the original design, or tears out good work, or relies on measurements that are now out of date. The benchmark tries to score that difference, which a plain pass/fail test on the original task would miss.

The caveat is that we have only the abstract. The details of the 12 change families, how each agent was set up, and how paired success is calculated sit in the full paper. These numbers describe this benchmark, not every real project.

What to do with this

For anyone choosing a coding agent, the lesson is to be cautious about a single leaderboard number, especially one from a frozen task. A fair trial includes a change partway through, such as a requirement tweak or a file edited by someone else.

For the people building these agents, the paper points to specific weak spots: switching tools or sessions, and noticing that earlier evidence has gone stale. Those are the places where, in this test, no system got a single paired success.