CTF-Dojo trains AI agents to find software flaws using 658 hacking puzzles
Researchers built a training ground of hacking challenges for AI agents, and a 32B open model reached 31.9% Pass@1, making bug-hunting AI easier to reproduce.
The short version
- CTF-Dojo has 658 working Capture-The-Flag challenges packaged in Docker for reproducible runs
- CTF-Forge, an automated pipeline, builds each environment in minutes instead of weeks of expert setup
- Agents were trained on only 486 execution-verified trajectories (recorded successful attempts)
- Gains reached up to 9.9% absolute over strong baselines on three benchmarks
- The best 32B model scored 31.9% Pass@1, a new open-weight record per the authors
Why it matters
Software bug-hunting is becoming something that smaller, openly available AI models can be trained to do, without relying on expensive proprietary systems. That helps defenders who audit code, and the same skill can help attackers. If you ship software, it is reasonable to expect automated vulnerability finding to get cheaper.
A practice arena for hacking agents
As arxiv.org reported, researchers have released CTF-Dojo, a collection of 658 working Capture-The-Flag challenges. A CTF is a hacking puzzle in which you find a flaw in a program to retrieve a hidden "flag", which proves you succeeded.
Each challenge is packaged in Docker, a tool that bundles software so it runs the same way on any computer. The authors say this guarantees reproducibility, so other teams can rerun the same tests.
Why a runnable environment matters
The paper's starting point is that language models improve when they learn inside environments where code actually runs. Success or failure is checked by the system itself, which gives a clear "verified" signal. This has worked well for software engineering tasks.
The authors say such environments have been scarce and hard to scale. A hacking challenge suits the approach, because it either yields the flag or it does not.
Building environments in minutes
To avoid hand-building every challenge, the team created CTF-Forge. It takes publicly available materials and turns them into ready-to-run environments in minutes. The authors say this removes weeks of expert configuration that were normally needed.
What the results show
The agents were trained on just 486 high-quality trajectories, meaning step-by-step records of attempts that were checked as successful by actually running them. The authors report gains of up to 9.9% over strong baselines on InterCode-CTF, NYU CTF Bench and Cybench.
Their best 32B model (32 billion adjustable settings) reached 31.9% Pass@1, the share of tasks solved on the first try. They say this sets a new open-weight best and rivals DeepSeek-V3-0324 and Gemini-2.5-Flash.
This is a preprint, a paper shared before formal peer review, and this version is a revised one. The numbers are the authors' own claims.
What to watch
- Will independent teams reproduce the 31.9% Pass@1 result once peer review is done?
- Do skills learned on puzzle-style challenges carry over to real-world software?
Try it
If you ship software, check whether your team has automated security testing in its build process, and assume attackers will soon have cheap automated tools of their own.
Sources and links
- Reportingarxiv.org: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by arxiv.org · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
