Planting 357 errors in real papers shows which AI reviewers actually catch mistakes
A new test hid known mistakes in real research papers. AI reviewers varied by up to 50 percentage points, and two models tuned for peer review found almost nothing.

The short version
- Researchers hid 357 known errors in 30 recent preprints: 15 in astrophysics, 15 in machine learning
- Detection differed by about 50 percentage points depending on the model and the prompt
- Skepthical caught 46.7% of astrophysics citation errors; unaided models caught 9% to 27%
- OpenReviewer-8B and PaperAudit-Bench Deep each found fewer than 3% of the planted errors
- Math critiques made up half of all false alarms on clean papers
Why it matters
Journals and conferences are deciding whether to let AI screen manuscripts. This test suggests the answer depends heavily on setup: a bare chatbot misses a lot, while a system that can run code and read cited papers does much better. The same lesson applies if you use AI to check contracts, reports or essays.
A test built on planted mistakes
On November 12, 2026, a team spanning Cambridge, the Flatiron Institute and a university in Barcelona posted a preprint on arXiv, the open server where researchers share papers before formal review. As CASRAI reported, the paper tackles a simple question: when an AI acts as a peer reviewer, how many real mistakes does it actually find?
Earlier evaluations often asked whether an AI's review resembled what human reviewers wrote, or whether people liked it. The authors took a different route. They inserted errors whose location and nature they already knew, so every catch and every miss could be scored objectively.
How the benchmark works
The test set was 30 recent preprints, all published after the training cut-off dates of the models being tested. That matters because a model cannot have memorised a paper it never saw. Fifteen papers were in astrophysics, with a median length of 13 pages, and fifteen were in machine learning, with a median of 9 pages.
Across these papers the team planted 357 errors in four groups: logical slips and contradictions between sections, faulty mathematical derivations, wrong numerical calculations, and bad citations. Each error was set at one of three difficulty levels: subtle, clear or blatant.
To keep the density of errors realistic, the planted mistakes for each paper were split across three separate copies. A fourth, untouched copy served as a control, so the researchers could count false alarms, meaning problems an AI flagged that did not exist.
What the 12 reviewers did
The team tested 12 systems. These included three frontier chatbots used with a plain prompt (Claude Opus 4.7, GPT-5.5 and Gemini 3.1 Pro Preview), Claude Code set up with a command line, Python and symbolic algebra tools, and three outside baselines: the Stanford Agentic Reviewer, OpenReviewer-8B and PaperAudit-Bench Deep. The last was the authors' own system, Skepthical, which uses several cooperating AI agents. It looks up references automatically, checks algebra with a symbolic maths library called SymPy, and runs Python code to verify numbers.
The first finding is that setup matters enormously. When each chatbot got the same short one-line request, the gap between the best and worst was about 50 percentage points. Giving more structured instructions, including a list of error types to look for, narrowed that gap. For Claude Code it also prompted the system to actually use its tools, which lifted its detection rate by more than 30 percentage points.
The second finding is that tools matter for specific kinds of error. On astrophysics citations, Skepthical found 46.7% of the planted errors by fetching and reading the cited papers. Models working from memory alone caught between 9% and 27%. On astrophysics numerical errors, Claude Code reached 91.1% by writing and running its own small checking scripts.
Where the tools fell short
The two models trained specifically on reviewing did worst. OpenReviewer-8B and PaperAudit-Bench Deep each found fewer than 3% of the planted errors in both fields. The authors read this as evidence that learning from past conference reviews teaches a model to write something that looks like a review, not to locate a technical flaw.
False alarms were a second weakness. Capable models flagged problems on fewer than 10% of clean copies overall, which sounds modest. But mathematical criticism was the shaky part: it accounted for half of all false alarms. An author who gets a confident but wrong comment about their equations pays for it in time and stress.
What this means for publishers and for you
The authors conclude that AI reviewers cannot yet replace human experts on the broader judgments, such as whether an idea is new, an experiment is well designed, or a result is important. They work best as narrow checkers of claims that can be verified. For editors, that argues against using a bare chatbot to triage submissions. Flags should not become automatic rejections, and human adjudication and clear confidence scores should stay in the loop.
Systems like Skepthical could serve as a pre-submission check for authors or as an extra layer when reviewing preprints and published work. Outside academia, the pattern is familiar. If you use an AI to read a contract, a grant application or a report, expect it to miss some real problems and invent others. Treat it as a second reader, and check its comments against the source.
One caution is the size of the test. It covers 30 papers in two fields, and the authors built one of the systems they tested. It is a useful map of strengths and weaknesses, not the last word.
What to watch
- Will journals and conferences adopt tool-equipped checkers, and will others reproduce these results on more papers and fields?
- Can the high false-alarm rate on math critiques be cut without losing real catches?
- Do independent teams test Skepthical, since its authors built both the system and the benchmark?
Try it
Next time you ask an AI to review a document, give it a list of the error types you care about, then verify five of its comments against the source yourself.
Sources and links
- PaperarXiv preprint auditing AI peer reviewers arxiv.org
- ReportingCASRAI: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by CASRAI · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
