A new benchmark finds AI memory gets worse at answering as it piles up old facts

Why memory tests miss the problem

As dev.to reported, ChatGPT, Claude and Gemini all now offer memory, with the same pitch: an assistant that carries your preferences, projects and goals from one chat to the next. OpenAI's own memory FAQ concedes a weakness. It says its earlier saved-memories system "often became stale" and could hold contradictory entries at once.

A paper called "From Recall to Forgetting" (Findings of ACL 2026, by researchers from Arizona State, the University of Arizona and Genies) argues that the tests used to grade agent memory share a blind spot. They check whether a system can retrieve a fact. Real long-term memory, the authors say, is mostly about revising facts.

The numbers back that up. In the LoCoMo benchmark, 94% of questions can be answered from at most two past sessions. In LongMemEval it is 85%. Updates are rare: LongMemEval allows at most two knowledge updates, and PersonaMem covers changes across no more than three sessions. So most scores measure whether a system can find something said once that never changed.

How Memora makes memory change

Memora, the new benchmark, simulates 10 professional personas, such as engineers, researchers, designers and executives. Each persona lives through a weekly, monthly or quarterly timeline. Those labels describe how long the history runs: about 155, 615 and 1,991 sessions per persona, at roughly 16 turns per session.

The key difference is that facts change. In the quarterly setting, a persona's memory goes through about 1,171 operations on average: 63% additions, 18% updates and 19% deletions. Every change is logged, so the benchmark always knows which facts are current and which are out of date when a question is asked.

Questions come in three kinds: remembering a fact, reasoning across several memories (such as working out how much budget is left from logged expenses), and recommending something that fits current tastes.

The scoring metric, FAMA (Forgetting-Aware Memory Accuracy), checks two things. Did the answer include what it should? And did it avoid what is no longer true? Using an invalidated memory costs points. Three AI judges vote on each criterion, and they agreed with human annotators 88.3% of the time on average.

What the tests showed

The researchers tested four language models reading raw conversation history: GPT-5.2, Claude Sonnet 4.5, Gemini 3 Pro Preview and Qwen3-32B. They also tested six dedicated memory systems: A-Mem, LangMem, Mem0, MemoBase, MemoryOS and Nemori. All six answered through the smaller GPT-4o-mini model, so this is not a like-for-like contest between memory and no memory.

One note on scale: the authors add scores across all three timelines, so totals run to 300 rather than 100. Treat them as relative scores, not percentages.

• Recall: memory systems nearly doubled the plain models, 119.5 versus 65.7.
• Recommending: plain models scored 144.7 to 153.3, against 138.4 for memory systems. The authors suggest guessing plausible tastes from context works well enough without precise retrieval.
• Reasoning: weak everywhere. The best average was 27.6, from the memory systems, while plain models sat at 12.4 to 13.9. In the authors' error review, each sampled mistake came from partial retrieval: the system had some pieces and answered anyway.

The cost of never letting go

The most telling result concerns stale facts. For plain models, the penalty for using outdated information shrank on longer timelines, from 32.6 to 17.8. But that is only because old conversations fell out of the model's context window, so it could not reuse them.

Memory systems went the other way. Their penalty grew from 18.2 to 29.5 as the stored history grew. The more they accumulated, the more they leaned on facts that had stopped being true.

That changed the rankings. In the monthly setting, MemoryOS (112.8) and A-Mem (112.0) beat Nemori (105.4) on plain accuracy. But Nemori's penalty was 15.4, against 28.4 and 29.5, so once stale answers counted, it came out ahead.

In the authors' sample of recommendation errors, 64% traced back to outdated memory that was never invalidated. One example: a user who had moved from non-fiction to contemporary fiction was still offered a historical biography. The system remembered, just not the current version of the person.

Caveats and what to take away

The conversations are simulated by AI, not gathered from real users. The authors argue that makes the results a lower bound, since real life adds more ambiguity. Even so, it is synthetic data. The study also covers only 10 personas and three kinds of memory: preferences, activities and goals. And the scoring relies on AI judges, though with majority voting and a human agreement check.

For builders, the advice is to save memories as versions that replace older ones, not as an ever-growing list, and to test with corrections and deletions. For everyone else, the direction is clear: more memory, badly maintained, can make answers worse.