SIAgentNews.com
← All news

An AI coding agent reviewing its own tests caught 0 of 6 planted bugs

A benchmark found a fresh, fault-hunting AI reviewer caught all six planted bugs where self-review caught none, for about half a cent more.

· First reported by dev.to

6 / 6planted bugs caught by the adversarial critic, versus 0 / 6 for self-review

The short version

  • Both setups passed all 20 hidden functional cases, so the code looked equally good.
  • Self-review wrote a test file that crashed on a missing tool and caught 0 of 6 planted bugs.
  • A fresh adversarial critic wrote a 50-line test file that caught 6 of 6.
  • The critic's extra pass cost +$0.0052 and +6.92s.
  • It is one run of one task on one model, so treat it as a strong hint, not a law.

Why it matters

If you use AI to write code, or to draft anything that needs checking, asking the same chat 'is this right?' may give you false comfort. A separate session told to find faults, plus a check you run yourself, is a cheap habit that this test supports.

What the experiment compared

As dev.to reported, a benchmark series called AI Labo tested two ways of checking the work of an AI coding agent. The test ran inside network-isolated virtual machines, so the agent could not reach the internet. The task was a small three-file program for a marketplace payment-holding and dispute system, using the model gemini-3.8-flash.

In the first setup, one agent wrote the code, then was asked in the same conversation to review its work, write extra tests and submit. In the second, a three-step chain was used: a maker wrote the code, a critic started with a fresh context and one goal (break the code), and the maker then finalized the work.

The harness, meaning the software that runs the agent, also locked the critic's test file as read-only. That stopped the maker from quietly weakening the tests to make them pass.

Same code score, very different safety net

On the hidden grading suite of 20 cases, both setups scored 20 out of 20. If that were the only measure, the two approaches would look identical.

The difference showed up in a second check called mutation testing. The grader planted six small, deliberate bugs, called mutants, into a clean copy of the program. A test file earns a point, or 'kills' a mutant, when it fails because of the bug. Examples included changing a 'greater than' into 'greater than or equal to', and accepting True as if it were the number 1.

The self-review test file killed 0 of 6. The critic's file killed 6 of 6.

Why self-review went wrong

The write-up points to two failures. First, the self-reviewing agent forgot a rule from earlier in the task: the sandbox had only Python's built-in tools. It imported a popular testing library, pytest, that was not installed. It then submitted without running the file again, so the test file crashed.

Second, even ignoring the crash, its tests were comfortable ones. It checked a dispute value of 101 but not the edge between 100 and 101. It tested one mid-sized order total of 13500, not the pair 7999 and 8000 where a rule flips. It left out a test for conflicting prices or rates on duplicate entries.

The author calls this confirmation bias: an agent that wrote the code carries its assumptions into the review, and so checks that things work rather than trying to make them fail.

What the critic did differently

The critic had no stake in the maker's code. Given a budget of 55 lines and a mandate to falsify, it wrote a 50-line test file using only built-in Python. It targeted the exact edges: 100 versus 101, 7999 versus 8000, True passed where a number belongs, and a required output format showing -0.

It also checked that a price of 0 is allowed, while -1 is rejected, and that conflicting duplicate entries raise an error. Its file ran cleanly and caught all six mutants on the first try.

The cost, and the caveats

The critic setup used 3 model calls instead of 2. Input tokens, the chunks of text the model reads, rose from 6,512 to 10,289. Run time went from 34.05 seconds to 40.97 seconds, and the estimated cost from $0.0213 to $0.0265. That is the +$0.0052 and +6.92s noted above.

The caveats matter. This is one task, one model and one run, and the post does not describe repeated trials. The idea of a separate reviewer is also an old one in software. What the post adds is a concrete measurement, and a design lesson: treat verification as its own component with its own incentive, not as a closing paragraph from the author.

What to watch

  • Whether the 0% versus 100% gap holds across repeated runs, other models and other tasks.
  • Whether agent tools start building in separate critic roles and read-only test files by default.

Try it

Next time an AI gives you code, a formula or a summary, open a new chat, paste in only the output, and ask it to find where it could be wrong, especially at the edge values. Then run it yourself.

Sources and links

First reported by dev.to. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news