SIAgentNews.com
← All news

AI red teaming may be training models on repetitive prompt injections

Researchers found that training AI agents on thousands of attacks can still leave them vulnerable if those attacks use the same underlying techniques.

· First reported by helpnetsecurity.com

47.5%Hijack rate of model trained on the larger attack pile

The short version

  • Automated attack tools often get stuck on a single trick, reusing it thousands of times with minor wording changes.
  • Models trained on larger piles of repetitive attacks performed worse in tests than those trained on smaller, diverse sets.
  • Researchers developed PIPETTE to reward tools for finding new techniques, significantly improving agent defenses.
  • Hardened agents using PIPETTE maintained high task completion rates without becoming overly timid or refusing legitimate work.

Why it matters

For developers, this means that a high 'attack count' from a red-teaming tool doesn't guarantee a secure model. It highlights the need to prioritize the diversity of attack techniques over the sheer volume of training data.

The illusion of security

Companies often try to harden AI agents against prompt injections by using automated red-teaming tools. These tools are designed to find ways to trick an AI into following hidden instructions, such as sending a payment or forwarding a file. Developers then collect every successful attack and retrain the model to recognize and block those specific prompts.

However, a study by researchers at the University of Waterloo, Zhejiang University, and KTH Royal Institute of Technology suggests this process might be flawed. They found that when an automated tool finds a trick that works, it tends to repeat that same trick over and over again. Even if the wording changes slightly, the underlying logic remains the same.

This creates a false sense of security for vendors who claim to have tested their models against thousands of attacks. As helpnetsecurity.com reported, the researchers found that a bigger pile of attacks actually led to a weaker defense in three out of four tests. The model essentially learned to recognize a few specific patterns while remaining wide open to different methods.

How the research was conducted

The research team set up a contest between two existing attack tools, PISmith and RL-Hammer, to target a small model called Qwen3-4B. They tested the tools across 560 different practice tasks to see which could break into the model more effectively. PISmith appeared to be the clear winner, breaking into 68.75% of the tasks and producing a haul of 7,132 working attacks.

The researchers then used reinforcement learning to retrain the model on each tool's collection of successful attacks. They then tested the retrained models against 40 new attacks that the tools had never seen before. These 40 tasks were specifically chosen because they were known weak spots that simple attacks could already break on an unprotected model.

The results were surprising. The version of the model trained on the much larger pile from PISmith was hijacked 47.5% of the time. In contrast, the model trained on the smaller pile from RL-Hammer was hijacked only 27.5% of the time. This suggests that the larger collection provided more copies of the same tricks rather than broader coverage of different vulnerabilities.

The problem of repetitive tricks

The researchers attribute this failure to the way automated attack tools operate. Once a tool discovers a successful strategy, such as telling an agent that a user's task is already finished, it tends to stick with that strategy. While both tools used checks to ensure variety, those checks only measured differences in wording, such as changing capital letters or adding irrelevant text.

Because the tools were not forced to find new techniques, the training data became saturated with variations of the same few prompts. The researchers noted that by their own measures, both tools stopped covering much new ground early in the training process. This means the agent was being reinforced on a narrow set of behaviors while the rest of the attack space remained unexplored.

This creates a significant hurdle for developers who rely on automated systems to find every possible way an AI might be manipulated. If the red-teaming tool is 'lazy' and only finds one way to break the system, the resulting training data will only teach the AI how to defend against that one specific way. This leaves the agent vulnerable to any attacker who thinks of a different method.

A new way to train agents

To solve this, the researchers built a system called PIPETTE. This system sorts successful attacks by the specific technique used and gives the attack tool an extra reward when it finds a new technique. By incentivizing variety, PIPETTE ensures that the training data covers a much wider range of potential vulnerabilities rather than just repeating the same few tricks.

The researchers also updated the agent's behavior. They retrained the agent to check, before taking any action, whether that action actually serves what the user originally asked for. This is a crucial distinction because standard hardening methods can sometimes make an agent too timid, causing it to quit whenever it sees data that looks even slightly suspicious.

The results using PIPETTE were significant. On a public test bed called AgentDojo, off-the-shelf injection attacks almost never worked against the PIPETTE-trained agents, succeeding in less than half a percent of attempts on average. Furthermore, the hardened models were not less useful; the Qwen3-4B model actually finished slightly more clean tasks than the original version.

Comparing defense methods

The study also compared PIPETTE to other common defense strategies, such as detection tools like PromptGuard. PromptGuard works by flagging and throwing out any email or document it deems suspicious. However, this can be problematic because if the tool flags a document, the agent loses the data it needs to complete the user's request.

In one test on the Qwen3-4B model, PromptGuard blocked slightly more than PIPETTE did, but the agent behind it only finished the user's real task 16.19% of the time. In contrast, the PIPETTE-trained agent finished the task 58.10% of the time. This highlights the difficulty of creating a defense that is both secure and helpful to the user.

The researchers tested models ranging from 4 billion to 27 billion parameters and found that while adaptive attackers could still occasionally get through, the PIPETTE method was highly effective. They noted that while a larger budget for an attacker might increase success rates, the diversity of techniques remains a primary factor in how well an agent can be defended.

Sources and links

First reported by helpnetsecurity.com. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news