Swapping a search model let one agent memory use 8x fewer tokens

What the developer changed

As dev.to reported, the developer behind Turnbook changed a single component in the tool and saw its token use drop sharply. Turnbook is a local memory for coding agents. It keeps a word-for-word record of conversations with six assistants, including Claude Code, Codex and Cursor, in one SQLite database file on your own computer.

When an agent needs something from the past, Turnbook searches that record and hands back a short bundle of evidence, which the author calls a packet. Saving the conversations calls no AI model, so storage costs no tokens. Tokens are only spent when the packet is read.

For version 0.10, the author replaced the search model, the part that decides which past passages look relevant. The old one was multilingual mpnet. The new one is EmbeddingGemma, a compact version that runs on an ordinary CPU. The author expected a small accuracy gain and instead got mostly a cost cut.

How the test was run

The benchmark was LongMemEval_S: 500 questions, each about a long chat history of roughly 115k tokens. The memory searches that history, builds a packet of limited size, and a separate reader model, gpt-5-mini, answers using only the packet. The benchmark's official gpt-4o prompt acts as the judge.

All 500 questions were counted. If the reader ran out of output budget while still thinking and never answered, that counted as a wrong answer. Only two things varied between rows: the search model and the packet size.

The results tell the story. With mpnet, accuracy was 86.6% at a median of 3.4k tokens, 89.6% at 6.7k, 90.4% at 13.2k and 91.8% at 26.3k. EmbeddingGemma hit 91.8% at 3.1k tokens.

At equal packet size, 33 questions were answered correctly only by the new model, against 7 that only the old one got right. The author reports an exact McNemar p of 0.00004. That is a statistical test for paired results, and a value this small means the gap is very unlikely to be chance.

Why a better search saves tokens

The author's explanation is about ranking. A weaker search model often does find the right passage, but places it 15th or 40th in the list. The only way to get it in front of the reader is to send a bigger packet, so accuracy is bought with tokens.

A stronger model places the right passage near the top. A small packet then already contains it, and there is no need to pad the prompt.

The biggest gain came on multi-session questions, which need facts scattered across several separate conversations. Accuracy there went from 73.6% to 88.4%. On LoCoMo conversations, which the re-ranker (a second model that reorders search results) had not seen, the answer appearing in the top 10 results rose from 70.2% to 83.7%.

Other effects, and the trade-offs

Other languages improved too. In meaning-only search with Korean questions, the answer landed in the top 30 results 86.9% of the time, up from 67.5%. A Korean question found an English conversation 87.8% of the time. The memory server also shrank from about 772 MB to about 525 MB.

The cost is speed. A search takes about 40 ms instead of about 26 ms on a GitHub runner, the cloud machine used for testing. The author calls that a fair trade but says plainly that it is one.

There are caveats on the headline number. The small re-ranker on top was trained on LoCoMo and LongMemEval data together, so 91.8% is not a clean score on unseen material. On the 380 questions never used in tuning, the score is 91.1%.

Licensing is another point. EmbeddingGemma is covered by Google's own Gemma Terms of Use, and Turnbook fetches it the first time you use it instead of bundling it. Existing users' memories are re-indexed slowly in the background, and until that finishes, searches still use the old model.

How much to trust it

This is a single developer's report on their own product, so treat it as a strong lead rather than a settled finding. In its favor, the author published every question, packet, answer and judge verdict in the repository under research/runs, so anyone can check a row.

The result also may not transfer. Your documents, your questions and your reader model may behave differently. The pattern is still worth knowing: a cheaper way to improve an agent's memory can be better search, not bigger prompts.

The author has asked others who have measured different embedding models for agent memory to share results, especially at small packet sizes. That kind of comparison would show whether the effect is general.