A provenance check can stop an agent's poisoned memory from firing again

An attack that outlives the attacker

Most research on prompt injection, where hidden instructions in text trick an AI into misbehaving, treats it as a single event. Agents with long-term memory and scheduled tasks break that picture. As dev.to reported in a post by a developer who built a defence, if the planted text gets saved to memory or turned into a recurring task, it can run again later.

The attacker plants it once and leaves. A later session recalls the note, or a timer fires the task, and the agent acts with no attacker present and no user watching.

The post points to a research paper, CarryoverBench (arXiv:2610.28425), which measured this and found planted payloads kept re-firing round after round without fading. That paper measures the attack but offers no defence, which is the gap the author set out to fill.

How the gate decides what runs

The core idea is to check where something came from, not what it says. When an entry is stored, the memory store itself writes a derivation record. The agent never writes it, because an agent's later account of an entry's origin cannot be trusted. The record includes the source channel, the author, links to parent entries, a policy version, and a hash (a fingerprint) covering the content and its parents' fingerprints.

From that record the store assigns one of four authority classes:

• User-authored.
• From a source the user approved.
• Derived by the agent, but only from the first three classes with an unbroken chain.
• Untrusted or unknown: web and tool content, pasted third-party text, unknown imports, broken chains, and anything derived from untrusted material.

When an entry is about to be reused, the gate looks at the record again. The first two classes run. The third can be read freely, but any side effect, meaning an action with consequences outside the chat, must fall inside a mandate the user pre-approved, or the gate asks the user.

The fourth class can be read, with a label telling the agent not to follow instructions in it. If it looks like an instruction or is about to cause a side effect, it is blocked and quarantined. A fingerprint mismatch or missing parent gets quarantined whatever the class. The gate is plain Python with no model calls, and a check took tens of microseconds (a median of 37 µs in one test, 53 µs in the other).

What the tests showed

The author ran two tests. The first was a small synthetic agent loop with a memory file and a task list. It held five planted memory payloads and five planted scheduled-task payloads, including a task from an unknown plugin, one tampered with on disk, and one built from pasted third-party text. Legitimate controls were mixed in. Each scenario ran five rounds with the gate off and on, using a stub agent that obeys anything and a small open model, qwen3-4b-instruct, run through Ollama.

For scheduled tasks, attacks went from 60% to 0% with the open model, and from 100% to 0% with the stub. With the real model that was 15 of 25 attack sessions down to 0 of 25. There were no false positives on the legitimate controls (0 of 605), and legitimate scheduled tasks finished at 97% either way.

The second test used Amazon's Task-Tampering benchmark (arXiv:2611.07640), 30 of its 90 cases, for 120 samples in all. Attack success moved from 20% to 13.3%, or 6 to 4 of 30. The author says that is within noise at this size. The author also committed the hypotheses in writing before the pilot ran and did not retune the gate afterward.

Where the gate falls short

The gate judges origin and intent to act, not truth. The sharpest example is a planted memory note falsely claiming a landlord's account number had changed. It is not phrased as an instruction, so the gate let it through in all five rounds, as the author had predicted. Most of the benchmark's injections work the same way, reading like plain facts such as a policy update. The gate withheld only 5 of 24 injection-bearing items. The author calls this a design limit, not something tuning would fix.

There are costs too. In clean benchmark runs, 13 of 30 samples (43.3%) saw some gate intervention. In the synthetic test, a third of benign untrusted entries were quarantined, which came down to one harmless note written like a request. The 0% false-positive figure applies only to the controls as the author defined them.

Two of the four hypotheses did not come out cleanly. One, about pairing the gate with the benchmark's other defences, was not tested. The no-injection control also failed to hold, since the model wrote an email agenda differently once the gate added labels. A bug that misread a shared ancestor as a loop mislabelled four blocked calls in the logs. It has been fixed and no outcomes changed.

How far to trust the numbers

This is a one-person pilot with small samples. The agent was a 4-billion-parameter open model. The judge scoring the benchmark was a local model, mistral-nemo-12b, that got 2 of 3 of the author's preflight probes right. So none of the results compare to the benchmark paper's published figures.

The benchmark covers a single session, so real cross-session re-firing was only tested in the synthetic harness. For agent-derived side effects, the benchmark stood in each case's list of required tools for the user's pre-approval, whereas a real product would have to get that approval from the person.

The author's own conclusion is narrow. If an attack depends on re-firing something the user never authorized, a quick provenance check at re-fire time can stop it. It is not a general defence against task tampering. The code, the written hypotheses and the run logs are public at github.com/sangaraju1988/carryovergate.