SI Agent News
DevelopingClaudeAgents

A 30-repo test shows how much data a coding agent's script can keep out of view

A dev.to test found one call per repo produced about 1.1 million tokens of output, while a hand-written script boiled similar data down to 627 tokens.

0.067%share of raw data the script passed on (25 repos)

The short version

Why it matters

If your AI assistant pulls data from many places, the raw material can overwhelm it before it starts thinking. Filtering in code first keeps the conversation small, but it also hides mistakes in the filter from the model. Check the filter itself, not only the summary.

What the test did, and what it faked

As dev.to reported, the authors ran a job that coding-tool users know well: checking a project's dependencies for trouble. On 2026-11-22, between 12:43 and 12:51 Singapore time, they fetched the newest issues from 30 popular open source repositories, one call each, through a remote data service called EveryInfra. That side of the test is real.

The Claude Code side is not. Code Mode is an experimental setting that lets Claude write a small JavaScript program to operate its tools. It never turned on here. The command the authors tried replied "Not logged in", and the feature only works in the interactive terminal anyway.

So the authors wrote a script by hand, following the documentation, and ran it in Node.js (a tool for running JavaScript on your own machine). Claude did not write it. Model token counts are also estimates from character counts, not measurements. The post is open about all of this, which is why its numbers deserve a fair reading rather than dismissal.

Thirty calls, a flood of text

The first round copied what an agent does by default: call one tool, receive the full result, read it, move on. The 30 results added up to 2,953,102 characters. Using a rough rate of 2.69 characters per token, the authors put that near 1.1 million tokens. A token is the small chunk of text a model counts as its unit of reading.

That total is larger than the 1 million token working memory of the Opus 5.5 model. The sizes varied a lot. The median result was 84,740 characters, TanStack/query was the largest at 323,135, and pmndrs/zustand was the smallest at 4,538 because it returned only 5 items, 3 of them pull requests.

Claude Code also has a safeguard for oversized tool output. In 28 of the 30 calls, a result passed the 50,000-character line where the tool stores output in a file instead of pasting it into the chat. Either way, the model still has to read it to use it.

How the script approach differs

Code Mode changes who does the sorting. Instead of the model reading each result, it writes a program that makes the calls, can run several at once, feeds one answer into the next request, and hands back only a short final object. You can still see the tool calls in the conversation. To view the program itself, press Ctrl+O and expand the collapsed Code row.

The feature arrived in Claude Code v2.1.323, published 2026-11-12. It is off by default and switched on with an environment variable, CLAUDE_CODE_EXPERIMENTAL_CODE_MODE=1. Project settings cannot enable it, so a repository you clone cannot turn it on for you. By 2026-11-22 the newest version was 2.1.330, and each release in between carried at least one Code Mode fix.

The post also notes a security wrinkle. Each tool call inside a script still passes through your deny rules and hooks, but the automatic reviewer checks the script only once, before it runs, so it cannot see values the script works out later. Strict limits belong in rules, not in that reviewer.

The result, and the catch

The hand-written script handled 25 repositories, reading 2,502,282 characters on the local machine. It kept only issues from the past 7 days that were not pull requests and whose titles or labels used words like regression, crash or security. What it returned was 1,687 characters, about 627 tokens, or 0.067% of the raw volume for those repositories.

The service fee did not depend on the method: $0.000958 per successful call, with rate-limited or failed calls free. Over the test, there were 41 attempts and 30 successes. Two later scripts made another 66 real calls.

Here is the caveat. The script cut the volume by more than 99%, yet its filter was wrong in 5 of 10 matches, and the model could not see that, because it only ever saw the summary. Smaller context is not the same as a correct answer.

How to read this write-up

This is a demonstration of scale, not a benchmark of Claude Code. It shows that the data is big enough to matter and that local filtering can shrink it a great deal. It does not show how well Claude writes, runs or fixes such a script on its own, and the authors say as much, suggesting readers run their own on-versus-off comparison.

What to watch

Try it

Anthropic's token counting endpoint is free. Paste a typical tool result from your own work into it with your model ID to see how large it really is before an agent reads it.

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by dev.to · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news