SI Agent News
DevelopingClaudeOpenAIGeminiAgents

Planting 357 errors in real papers shows which AI reviewers actually catch mistakes

A new test hid known mistakes in real research papers. AI reviewers varied by up to 50 percentage points, and two models tuned for peer review found almost nothing.

Ink and watercolour: picture for the story 'Study audits AI peer reviewers on 357 injected scientific errors'
AI-made ink and watercolour for this story. No real people or products are shown.
357known errors planted in 30 real preprints

The short version

Why it matters

Journals and conferences are deciding whether to let AI screen manuscripts. This test suggests the answer depends heavily on setup: a bare chatbot misses a lot, while a system that can run code and read cited papers does much better. The same lesson applies if you use AI to check contracts, reports or essays.

A test built on planted mistakes

On November 12, 2026, a team spanning Cambridge, the Flatiron Institute and a university in Barcelona posted a preprint on arXiv, the open server where researchers share papers before formal review. As CASRAI reported, the paper tackles a simple question: when an AI acts as a peer reviewer, how many real mistakes does it actually find?

Earlier evaluations often asked whether an AI's review resembled what human reviewers wrote, or whether people liked it. The authors took a different route. They inserted errors whose location and nature they already knew, so every catch and every miss could be scored objectively.

How the benchmark works

The test set was 30 recent preprints, all published after the training cut-off dates of the models being tested. That matters because a model cannot have memorised a paper it never saw. Fifteen papers were in astrophysics, with a median length of 13 pages, and fifteen were in machine learning, with a median of 9 pages.

Across these papers the team planted 357 errors in four groups: logical slips and contradictions between sections, faulty mathematical derivations, wrong numerical calculations, and bad citations. Each error was set at one of three difficulty levels: subtle, clear or blatant.

To keep the density of errors realistic, the planted mistakes for each paper were split across three separate copies. A fourth, untouched copy served as a control, so the researchers could count false alarms, meaning problems an AI flagged that did not exist.

What the 12 reviewers did

The team tested 12 systems. These included three frontier chatbots used with a plain prompt (Claude Opus 4.7, GPT-5.5 and Gemini 3.1 Pro Preview), Claude Code set up with a command line, Python and symbolic algebra tools, and three outside baselines: the Stanford Agentic Reviewer, OpenReviewer-8B and PaperAudit-Bench Deep. The last was the authors' own system, Skepthical, which uses several cooperating AI agents. It looks up references automatically, checks algebra with a symbolic maths library called SymPy, and runs Python code to verify numbers.

The first finding is that setup matters enormously. When each chatbot got the same short one-line request, the gap between the best and worst was about 50 percentage points. Giving more structured instructions, including a list of error types to look for, narrowed that gap. For Claude Code it also prompted the system to actually use its tools, which lifted its detection rate by more than 30 percentage points.

The second finding is that tools matter for specific kinds of error. On astrophysics citations, Skepthical found 46.7% of the planted errors by fetching and reading the cited papers. Models working from memory alone caught between 9% and 27%. On astrophysics numerical errors, Claude Code reached 91.1% by writing and running its own small checking scripts.

Where the tools fell short

The two models trained specifically on reviewing did worst. OpenReviewer-8B and PaperAudit-Bench Deep each found fewer than 3% of the planted errors in both fields. The authors read this as evidence that learning from past conference reviews teaches a model to write something that looks like a review, not to locate a technical flaw.

False alarms were a second weakness. Capable models flagged problems on fewer than 10% of clean copies overall, which sounds modest. But mathematical criticism was the shaky part: it accounted for half of all false alarms. An author who gets a confident but wrong comment about their equations pays for it in time and stress.

What this means for publishers and for you

The authors conclude that AI reviewers cannot yet replace human experts on the broader judgments, such as whether an idea is new, an experiment is well designed, or a result is important. They work best as narrow checkers of claims that can be verified. For editors, that argues against using a bare chatbot to triage submissions. Flags should not become automatic rejections, and human adjudication and clear confidence scores should stay in the loop.

Systems like Skepthical could serve as a pre-submission check for authors or as an extra layer when reviewing preprints and published work. Outside academia, the pattern is familiar. If you use an AI to read a contract, a grant application or a report, expect it to miss some real problems and invent others. Treat it as a second reader, and check its comments against the source.

One caution is the size of the test. It covers 30 papers in two fields, and the authors built one of the systems they tested. It is a useful map of strengths and weaknesses, not the last word.

What to watch

Try it

Next time you ask an AI to review a document, give it a list of the error types you care about, then verify five of its comments against the source yourself.

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by CASRAI · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news