A new paper argues AI agents do better searching raw history than using designed memory

Two ways to give an agent a memory

AI agents that work across many steps or sessions need some way to remember what happened earlier. One camp, which includes products such as Mem0 and Zep, builds memory ahead of time. The system decides which facts are worth saving before anyone asks a question.

The other camp is what coding agents do. They store the whole raw record and let the model hunt through it using its own search tools, the way a programmer greps through files. As arxiv.org shows in a new preprint called "MemoryTax," the authors wanted to know whether the first approach still earns its keep once the model is capable enough to do the digging itself.

The title nods to the "bitter lesson," a familiar idea in AI that general methods tend to beat hand-crafted tricks as models improve. Here the hand-crafted trick is the carefully designed memory layer.

What the researchers actually ran

The test used a single conversation from LoCoMo, a set of long chat logs paired with quiz questions. That conversation has 419 turns and 152 questions. The team compared four memory methods: full context (pasting the whole history into the prompt), BM25 (a classic keyword-matching search), Mem0 and Zep.

They also ran three coding agents, called Pi, Codex and Claude Code. Every one of these setups was paired with three models: GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5. For each complete run, the authors tracked the cost as well as the accuracy.

The authors say plainly that this is a short study with one run per combination. That limits how far anyone should stretch the results.

A patch for weak models, with a ceiling

The first finding is about cost and benefit. Mem0 came out ahead of full context only when paired with GPT-4o-mini, the model the authors treat as the weak one. Even then, the price of building Mem0's memory was higher than the price of an entire full-context run. The authors call designed memory a costly patch for a weak model.

The second finding is about limits. A memory system is written by a model, and whatever that writer leaves out is gone. Handing the memory to a stronger reader later does not bring back the missing details. The quality of the memory is capped by the model that wrote it.

The best result went the other way. With Claude Opus 5.5, Pi sat at the top of the cost-versus-accuracy trade-off, scoring 99% for less than half the cost of full context. The authors say its single miss was down to a wrong answer key, not a wrong answer from the agent.

The photo caveat and other limits

Some of Pi's lead has a plain explanation. The conversation included photos, and the agents could read the captions attached to them, which the other methods could not. That gave the agents access to evidence the others never saw.

When the authors looked only at the 88 questions that did not rely on a photo, the gap nearly vanished. Pi and full context differed by just one answer. So the clean version of the claim is narrower: for a strong model, searching raw history matches full context on these questions while costing far less.

There are other reasons for caution. The study covers one conversation, each setup ran once, and the paper is a preprint, meaning it has not been through peer review. It is a useful data point, not a verdict.

What builders can take from it

The practical message is about ordering. Before adding a memory layer that picks facts in advance, try keeping the raw history and giving the model good search tools. Measure both accuracy and cost, because the paper shows the build step alone can outweigh the savings.

It also points to a risk that is easy to miss: any pre-selection step bakes in the judgment of whichever model did the selecting. If that model was weak, upgrading the model that answers questions later will not fix it.

None of this proves designed memory is useless. It suggests the case for it gets weaker as models get stronger, and that teams should check whether their own tasks look like this test before assuming it holds.