Filtering test logs before a coding agent sees them cut input tokens by 94.1%

The hidden cost of a test log

As dev.to reported, a developer running a series of coding-agent benchmarks found that one of the biggest sources of wasted tokens is not the code. It is the output of the test runner. A token is the small chunk of text an AI model reads and is billed for, so more text in means a higher bill.

In the experiment, the test suite had 80 tests across three modules of a payment settlement program. Seventy-six passed and four failed. A standard pytest run (pytest is a common Python testing tool) printed all of it: color codes, environment banners, a line for every passing test, debug text and long tracebacks, which are step-by-step records of where an error happened.

That first run produced 60,749 characters. The author estimates a single run like this adds more than 28,000 tokens to the agent's conversation.

Two setups, one task

The author compared two ways of wiring the same agent to the same tests. The model was gemini-3.8-flash with its thinking level set to low, and it ran inside an isolated virtual machine with no network cards. Each setup was then graded on a hidden suite of 20 cases.

In the first setup, the raw terminal output went straight into the conversation. In the second, a small program sat between the agent and the test runner. It stripped the color codes, the banners, all 76 passing tests and the internal framework steps, then returned a short summary in JSON, a simple structured text format.

That summary was 1,506 characters. It gave the totals (80 run, 76 passed, 4 failed) and, for each failure, the test name, file, line number and a description of what was expected.

How the raw log snowballed

The trace shows why the raw approach got expensive. Every tool result stays in the conversation, so each new turn re-reads everything that came before. The first call used 2,406 input tokens. By the second, after the 60,749-character log was added, input jumped to 31,170.

According to the write-up, the agent was distracted by the wall of text and patched only one of the three files. That triggered a second test run, which added another 56,544 characters. The third turn then needed 58,809 input tokens, and the cumulative input total reached 92,385.

With the filtered summary, the agent saw all four failures at once. It fixed all three files in a single edit turn, which needed 3,030 input tokens. The whole job took two model calls instead of three.

The numbers side by side

• Cumulative input tokens: 92,385 down to 5,445, a 94.1% cut.
• Estimated cost: $0.0770 down to $0.0100, a drop of 87.0%.
• Elapsed time: 25.94 seconds down to 16.88 seconds, 35.0% less.
• Output tokens: 2,068 down to 1,571, a 24.0% reduction.
• Hidden test score: 20 out of 20 in both setups.

The last line matters most. The filtered agent was not cheaper because it did less or got worse results. Both finished the job correctly. It was cheaper because it avoided reading text it did not need.

What to take from it, and what to doubt

The author's advice is simple: do not pipe raw test output into an agent. Use pytest's quieter options or a JSON reporter plugin, remove color codes and passing tests, and pass along only the failure details plus a pass count. The author also points to a tool called rtk (Rust Token Killer), or a short Python wrapper, and claims savings of 85% to 95%.

Keep the limits in mind. The write-up describes one run, one model and one small, purpose-built project, run by the author. It is not an independent study, and your results will depend on how noisy your own test output is. A project with few passing tests and short logs will save much less.

Still, the logic holds up in plain terms. An assistant given a short, clear list of what is broken has less to wade through than one given a long transcript where the problems are buried.