Signing AI agents' requirements stops tampering, but not mistakes made before signing
A preregistered study finds signed requirement lists sharply cut unsafe outputs when agents hand off work, but the weak point is how the list is first written.
The short version
- Under attack, signed manifests cut unsafe commits from 69.4% to 9.0% on Qwen3-8B (1,440 runs, five threats).
- An unsigned manifest committed violations in 79% of tampered runs; the signed one in none.
- Code-based checking caught 78.7% of violating decisions; an AI checker caught 59.0%, with far more false alarms.
- One adversarial line added before signing pushed violations as high as 75.0% on GPT-5.6-luna.
Why it matters
Many teams now chain AI agents, with one planning, one drafting and one merging. This study suggests a sign-off at the end is only as good as the first step that turned your request into a checklist. If you build or buy agent workflows, ask who writes that checklist and who can influence it.
The problem: requirements get lost in handoffs
Companies increasingly deploy AI agents that split a user's request into pieces, send the pieces to helper agents, and merge the results. As arxiv.org reported, every handoff is a chance for a requirement to be shortened, altered or quietly dropped.
Think of an instruction like "never promise a refund date." One agent paraphrases it, the next never sees it, and the final answer breaks the rule.
The fix the researchers tested
The main agent (the orchestrator) pulls out each requirement as an exact quote plus a checkable rule. Ordinary code then signs that list with HMAC-SHA256, a standard method that makes any later edit detectable. At the end, plain code, not another AI, checks whether the merging agent's decision obeys the list.
The team compared this with free-text handoffs and structured JSON handoffs, using synthetic tasks that could be scored exactly. With the small Qwen3-8B model, signed lists cut unsafe commits (outputs accepted despite breaking a requirement) under attack from 69.4% to 9.0%, without losing valid ones compared with structured handoffs.
Why the signature and the code checker matter
Signing is what stops tampering in transit. An unsigned list still led to violations in 79% of altered runs, while the signed one led to none.
The code-based checker also beat an AI checker on the same runs. It caught 78.7% of violating decisions versus 59.0%, and raised false alarms 7% of the time versus 50%.
The catch: garbage in, signed garbage out
When the researchers handed the system a perfect list, the remaining unsafe commits vanished. So the leftover risk comes from mistakes in the extraction step, before signing.
A second test confirmed it. A single adversarial line planted before signing raised violations from 14.6% to 37.5% on Qwen3-8B, from 11.1% to 63.9% on gpt-oss-20b, and from 8.3% to 75.0% on GPT-5.6-luna. The signature then vouched for the corrupted list. The extraction step is where trust has to be earned.
One caution: the tasks were synthetic, so real-world results may differ.
What to watch
- Do the results hold on messy real-world tasks rather than synthetic ones?
- How can the extraction step be protected against planted instructions?
Try it
If you run a multi-agent workflow, have a person review the extracted requirement checklist before the work starts, and check the final output against it with code where you can.
Sources and links
- Reportingarxiv.org: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by arxiv.org · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.