Anthropic test: AI agents improve science models, but mostly with routine engineering

What the study actually found

As 36Kr reported, a team at Anthropic built a test called Agent4Sci to ask a narrow question: can an AI coding agent, left alone, make a machine learning model for a science problem better? A coding agent is an AI system that reads code, edits it, runs it and checks the results by itself.

The headline conclusion is that AI still cannot do genuine scientific research. The details are more nuanced. Four leading agents together improved all 20 benchmarks, which are standardized tests with a fixed scoring method. Claude Opus had the best average relative improvement, 10.4%. That means its scores rose by about a tenth compared with where they started.

The catch is where the gains came from. When the team studied the agents' step-by-step logs, they concluded the improvements came mainly from conventional machine learning engineering tricks, not from research-level insight.

How the test keeps agents from cheating

Earlier experiments showed a problem. If you let an agent loose on a raw research codebase with vague scoring rules or unclear data splits, it can find loopholes and post better numbers without making the model better. So the team built guardrails.

First, they chose tasks carefully. Using the NeurIPS 2025 Data and Benchmarks track as an example, they picked 10 benchmarks from 497 candidate papers, a rejection rate of about 98%. Each chosen task had to reproduce the original paper's baseline, and its data splits, environment and training process were standardized by hand.

Second, they split each run into three isolated roles:

• Climbing agent: gets the papers, code, training data and validation data, then edits code and trains models within a fixed budget. It never sees the test set.
• Evaluation agent: separately computes final scores on held-out test data, meaning data kept hidden from the climber.
• Auditing agent: reads the logs and outputs, looking for score gaming and other misconduct.

Auditors from the Claude, Codex and Gemini families cross-checked part of the runs, agreed with one another and flagged none. The authors still caution that this does not rule out every hidden trick.

Four agents, one budget, uneven gains

The agents were Claude Opus 4.6, Claude Sonnet 4.6, Gemini 3.1 Pro and Codex (GPT-5.4), each run through its official command-line tool at the highest reasoning setting. Each run got 8 H200 GPUs, which are high-end AI chips, and a 12-hour budget. No human made decisions during the runs.

Average relative improvements were 10.4%, 9.6%, 6.8% and 4.4%, in the order the models are listed. Not every agent improved every benchmark, but at least one improved each of the 20. Individual gains ranged from 1.1% to 64.4%.

Working styles differed too. Opus kept watching its logs and iterating. Codex tended to launch long scripts early and then wait. Gemini fell between the two.

Tuning is not the same as understanding

The standout case was PFDelta, a power-grid estimation task, where Opus cut the average power balance loss by 64.4% by adjusting auxiliary loss terms, raising the weight on one error measure and changing the learning-rate schedule. On a Mars-Bench task that detects cones in images, Opus swapped the original detection model for Faster R-CNN, which suits the task better, and lifted its mAP score from 16.5 to 30.4.

Those are real gains, but they look like skilled engineering. Across the logs, agents leaned on well-known moves: adjusting learning rates, making networks deeper or wider, replacing the backbone and adding weight decay. Changes aimed at the specific science problem were comparatively rare.

The team then wrote three lightweight add-on skills, covering research ideation, experiment management and timed checklists. With them, Opus's average gain rose to 20.1%, and it improved 18 of 20 benchmarks. PFDelta went from 64.4% to 91.0%, MassSpecGym reached 39.5% and CarbonGlobe 35.3%.

The new PFDelta approach looked more like physical reasoning. The agent noticed that a loss term can be derived from Kirchhoff's law, a basic rule of electric circuits, so it needs no labeled data. It rebuilt its main training objective around that. So the picture is mixed: agents can sometimes reach for domain knowledge, but mostly they do not.

What this means for readers

The study measures one thing: improving models on tasks that already have code and scoring. It does not test choosing a research question or running physical experiments. It is also reported secondhand here, so read the original paper for the full detail. The work was selected for the Evaluation and Datasets track at NeurIPS 2026.

The authors say the open question is whether coding agents can move from running experiments efficiently to proposing more valuable research ideas. For now, treat agents as fast, tireless engineers who can polish results, while people decide what is worth asking and what a result means.