DKDKCISSP
AI SecurityINCIDENT

Your AI red team may be finding the same prompt injection thousands of times

Researchers found that AI agents retrained on prompt injection attacks became easier to hijack, raising concerns about attack tool effectiveness.

DKCISSP News Desk20 Nov 2026, 12:00 pm2 min read
Your AI red team may be finding the same prompt injection thousands of times
Image courtesy of Help Net Security. Original report

Researchers from the University of Waterloo, Zhejiang University, and KTH Royal Institute of Technology discovered that AI agents retrained on a larger set of prompt injection attacks were more susceptible to hijacking in three out of four tests. This finding highlights potential weaknesses in the effectiveness of current automated red-teaming tools.

In a study examining the security of AI agents, researchers found that models retrained on a larger volume of prompt injection attacks performed worse in defending against new attacks. The automated red-teaming tool, which generated more successful attacks, ultimately resulted in a weaker defense for the AI agent. This raises important questions regarding the quality of the training data used to secure these models.

The researchers conducted a contest using two attack tools, PISmith and RL-Hammer, targeting the same small AI model, Qwen3-4B, across 560 practice tasks. PISmith achieved a success rate of 68.75%, significantly outperforming RL-Hammer, which had a success rate of 33.39%. PISmith returned 7,132 successful attacks compared to RL-Hammer's 4,282. This stark difference underscores the variability in attack effectiveness based on the tool used.

Prompt injection attacks involve embedding instructions within inputs that AI agents process, such as emails or web pages. When an AI agent interprets these inputs, it may inadvertently follow harmful instructions, like transferring funds or sharing sensitive data. To enhance defenses, developers often use automated tools to identify successful attacks and retrain the AI agents on those attacks, hoping to improve resilience against future threats.

The impact of this research is significant, as it suggests that AI agents retrained on repetitive attack patterns may not be adequately prepared for diverse and evolving threats. The study's findings indicate that relying on a large volume of similar attacks could lead to vulnerabilities, as the AI may only learn to defend against slight variations of the same attack.

The timeline of the study does not specify exact dates for each phase of the research, but it outlines a systematic approach where the team first tested the AI agent against different attack tools, then retrained it, and finally evaluated its performance against new attacks. This structured methodology highlights the rigorous nature of the research.

The researchers have proposed a new system called PIPETTE, which categorizes successful attacks by technique and incentivizes the discovery of new attack methods. This approach aims to enhance the training process by ensuring that AI agents learn from a broader range of techniques, rather than just variations of the same attack.

Unknowns in this research include whether the findings will hold true across different models and configurations, as well as how these results will influence the development of future AI security measures. Additionally, the researchers have not yet evaluated how their PIPETTE system performs against human judgment in assessing technique coverage.

DKCISSP VIEWOur take, not reported fact

What it meansThe study shows that AI agents retrained on similar prompt injection attacks may become less effective at defending against new threats.

Ask your IT team this weekHow diverse is our training data for AI models?What measures do we have in place to evaluate the effectiveness of our AI defenses?

DecisionThe IT team reviews and diversifies the training data used for AI models, and security checks the implementation.

MORE IN AI SECURITY

More cybersecurity reporting

AI agent API security hasn't caught up with agents already running in production20 Nov 2026, 11:00 amPalo Alto Networks to acquire Applied Compute for $4.5 billion19 Nov 2026, 3:53 pmAI agents inherit employee access, and nobody's watching how they use it19 Nov 2026, 10:30 amQuality Isn’t The Main Worry About AI Pentesting — Data Privacy and Production Safety Are18 Nov 2026, 4:33 pm