Locking down a medical AI agent cut attacks, but diagnosis tampering still got through
A healthcare study found that giving an AI agent fewer permissions blocked most data theft, yet attacks that change a diagnosis still worked.
The short version
- RAG poisoning success fell from 31% to 10% after the redesign.
- Data-field injection success fell from 42% to 25%.
- Report accuracy rose from 72.5% to 86.5%.
- No patient record was disclosed under the redesign.
- 32 of the 35 successful attacks on the redesign altered the Covid-19 diagnosis.
Why it matters
Many organisations give AI agents broad access so they work better, and this study shows that choice carries a measurable security cost. It also shows the limit of the fix: narrowing what an agent can reach does not stop it from being fooled inside its own task. For anything high-stakes, a human or a separate check must verify the output.
What the researchers did
As arxiv.org reported, a research paper took a medical report-generation app and rebuilt it using TRiSM, short for AI Trust, Risk, and Security Management. TRiSM is a framework for controlling how AI systems are given access and how they are monitored. The authors began with an insecure baseline and turned it into a security-conscious version of the same workflow.
The starting problem is common. To make agents more capable and accurate, builders often hand them access rights beyond what an ordinary user would have. That raises the odds of security incidents and of breaking regulations, which matters a great deal when the data is medical.
The test: three ways to attack an agent
Both versions were run with five language models, including Claude Haiku 4.5 and Gemini 2.5 Flash, plus three from the GPT family. The study covered 800 report generations and 500 attack trials.
The attacks came in three types:
- RAG poisoning: planting misleading text in the documents the agent looks up while writing.
- Data-field injection: hiding instructions inside the data the agent reads, such as a record field.
- Client-side network injection: tampering with the network traffic on the user's side.
What improved
The redesign helped on several fronts. RAG poisoning succeeded 31% of the time against the baseline and 10% against the new design. Data-field injection dropped from 42% to 25%. The authors report both drops as statistically significant (McNemar p < 0.001, a test for comparing paired results).
Network injection was handled differently: the new architecture made it impossible by design, so there was nothing left to measure. Report accuracy also went up, from 72.5% to 86.5%, so the safer workflow was not a trade against quality here.
The authors then grouped attacks by what they were trying to achieve. They describe this as exploratory and done after the fact, so treat it with some caution. Attacks that pushed the agent toward goals unrelated to writing the report, like leaking data or misusing tools, succeeded 21.7% of the time before and 2.5% after. Under the redesign, no patient record was disclosed.
The attack that still worked
The weak spot was attacks that changed the Covid-19 diagnosis stated in the report. They made up 32 of the 35 successful attacks against the redesigned workflow.
The authors' explanation is clear. A least-privilege design, meaning the agent can touch only what its job needs, limits how far an injected instruction can reach. It does not stop a model from believing planted content when that content sits inside the task it is already doing. A report that wrongly states a diagnosis is a failure that the permission model cannot see.
Their conclusion is that clinical outputs need verification that does not rely on the same language model. A second check, such as a rule-based comparison against the source record or a clinician's review, would be independent in the way they mean.
Picking a model is not a defence
The paper also reports that none of the five models resisted all three attack types. That means choosing a supposedly safer model cannot replace building the workflow securely. The structure around the model did more of the work than the model's name.
For readers outside healthcare, the pattern carries over to any agent handling sensitive records: decide what it may access, keep that list short, and assume the content it reads could be hostile. Then add a separate check on the results that would hurt most if wrong.
What to watch
- Whether other teams reproduce these results in other medical or regulated settings.
- What independent verification of clinical outputs looks like in practice, and whether it catches diagnosis tampering.
- Whether this exploratory attack-goal grouping holds up under a pre-planned test.
Try it
List what each AI tool you use can access, then remove anything it does not need for its job. For any output that matters, add a check that does not rely on the AI itself.
Sources and links
- Reportingarxiv.org: the article we sourced
First reported by arxiv.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.