SI Agent News
DevelopingGeminiOpen modelsAgentsSecurity

CTF-Dojo trains AI agents to find software flaws using 658 hacking puzzles

Researchers built a training ground of hacking challenges for AI agents, and a 32B open model reached 31.9% Pass@1, making bug-hunting AI easier to reproduce.

31.9%Pass@1 score of the best 32B model

The short version

Why it matters

Software bug-hunting is becoming something that smaller, openly available AI models can be trained to do, without relying on expensive proprietary systems. That helps defenders who audit code, and the same skill can help attackers. If you ship software, it is reasonable to expect automated vulnerability finding to get cheaper.

A practice arena for hacking agents

As arxiv.org reported, researchers have released CTF-Dojo, a collection of 658 working Capture-The-Flag challenges. A CTF is a hacking puzzle in which you find a flaw in a program to retrieve a hidden "flag", which proves you succeeded.

Each challenge is packaged in Docker, a tool that bundles software so it runs the same way on any computer. The authors say this guarantees reproducibility, so other teams can rerun the same tests.

Why a runnable environment matters

The paper's starting point is that language models improve when they learn inside environments where code actually runs. Success or failure is checked by the system itself, which gives a clear "verified" signal. This has worked well for software engineering tasks.

The authors say such environments have been scarce and hard to scale. A hacking challenge suits the approach, because it either yields the flag or it does not.

Building environments in minutes

To avoid hand-building every challenge, the team created CTF-Forge. It takes publicly available materials and turns them into ready-to-run environments in minutes. The authors say this removes weeks of expert configuration that were normally needed.

What the results show

The agents were trained on just 486 high-quality trajectories, meaning step-by-step records of attempts that were checked as successful by actually running them. The authors report gains of up to 9.9% over strong baselines on InterCode-CTF, NYU CTF Bench and Cybench.

Their best 32B model (32 billion adjustable settings) reached 31.9% Pass@1, the share of tasks solved on the first try. They say this sets a new open-weight best and rivals DeepSeek-V3-0324 and Gemini-2.5-Flash.

This is a preprint, a paper shared before formal peer review, and this version is a revised one. The numbers are the authors' own claims.

What to watch

Try it

If you ship software, check whether your team has automated security testing in its build process, and assume attackers will soon have cheap automated tools of their own.

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by arxiv.org · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news