Five agent harnesses took one memory test, and scores ranged from 26.7% to 73.3%

Remembering is not the same as learning

As thenewstack.io reported, Letta has released a benchmark called FrontierMemory that targets a gap in how agents are judged. Many existing memory tests check whether an agent can recall a stored fact. Letta's CEO, Charles Packer, argues that this kind of recall is largely solved for modern coding agents.

The harder question is whether stored information makes the agent better at its next job. Packer's stated premise is that "if AI-generated memory is good, it should actually improve downstream task performance." A harness, for readers new to the term, is the software around a model that manages how the agent runs, including what it keeps in view.

Letta co-founder Sarah Wooders adds a second complaint. Most benchmarks hand an agent every piece of context up front and use unrelated tasks. Real work is different. A new hire needs lots of direction, while a veteran can do the same job with a short note.

How the test works

Agents in FrontierMemory are given transcripts of earlier work, including attempts that succeeded and ones that failed. From those, the agent writes its own memories. It then has to use them on later tasks. Letta reworked existing coding and knowledge-work tests to build this, and after fixing problems in the originals it kept 15 tasks for version 1.0.

The learning here happens in what Packer calls token space. That means the agent changes the notes and context it carries, not the model's internal weights, which are the numbers set during training. Developers already build agents this way, pairing a language model with memory systems and feedback loops.

The harness gap

Letta Dream, the company's own harness, scored 73.3%. Prime Agent reached 60%, OpenAI Agents SDK 40%, Claude Managed Agents 33.3% and Hermes Agent 26.7%.

In one matched comparison, Dream beat the OpenAI Agents SDK 60% to 40%. Both used a GPT model to write the memories and a slightly different GPT model to do the tasks. With Claude Opus 5.5 involved, the spread was wider still.

Packer credits much of the difference to how a harness handles "dreaming", meaning the step where an agent reviews past work and turns it into reusable instructions. He also flags a risk: with no checks on what gets saved, an agent can lock in bad habits and repeat them.

Which model writes the best memories

Letta also compared seven models on how useful their memories were. Claude Opus 5.5 and Z.ai's GLM-5.3 tied at 73.3%. Kimi K3 got 66.7%, and OpenAI's GPT-6.1 Sol and Google's Gemini 4 Argon each got 60%. Muse Wonder scored 46.7% and Grok 4.7 scored 33.3%.

Cost cut the other way. GPT-6.1 Sol came in at an estimated $0.77 per task. Opus 5.5 ran about $2.80 and GLM-5.3 about $3.58. So the cheapest option scored lower, and the top scorers cost several times more.

Letta's examples show why quality differed. In a pricing-data task, Opus 5.5 wrote one skill saying when to use certain information. GPT-6.1 Sol wrote four separate skills, and the agent never located the one it needed. Gemini 4 Argon saved particular file sizes, which were useless once the files changed. Opus 5.5 instead advised checking the usual file and also email, and that held up when the task shifted.

Limits and what comes next

Memories written by Opus 5.5 also helped GPT-6 Sol finish tasks. That hints at a practical split: pay for a stronger model to write the notes, then let a cheaper one do the daily work.

The caveats are real. With 15 tasks and a single attempt on each, one task can swing a score by roughly 6.7 percentage points. The benchmark also looks only at learning from finished work, not at agents updating memory as they go. Letta says a later release will bring longer tasks and live memory updates.

Outside interest has started. The OpenClaw account asked for a plugin, and Wooders replied she would like to add it, calling memory shared among several agents an interesting future direction.