Showing a coding agent only function outlines beat giving it full source code

What the benchmark tested

As dev.to reported, an author running a series of coding-agent experiments compared two ways of preparing an AI agent for a refactoring job. The task involved a small payment settlement pipeline made of three Python files: validator.py, settlement.py and statement.py. The agent had to edit these while respecting how they connect to other code.

In the first setup, called Mode A, the agent was given the full source of the files it depended on, plus raw diagnostic output. That added up to 78,719 characters of tool output over three turns. In the second, Mode B, the agent got only an outline of the files, with no implementation details.

Both ran inside isolated virtual machines with no network cards, and a hidden suite of 20 test cases did the grading. The model was gemini-3.8-flash, set to low thinking effort and temperature 0.0.

What an interface skeleton is

The outline in Mode B is called an interface skeleton. A tool called Tree-sitter reads code and builds a structured map of it (an abstract syntax tree, or AST). The harness then keeps each function's name, inputs, types and its written contract, and replaces the body with three dots.

The contracts were specific. For example, one said a unit price of zero is allowed, and another said a chargeback threshold applies only when it is strictly above 300 basis points. These are the edge cases where an agent can easily slip.

The skeleton added only 673 tokens on the first turn. Tokens are the chunks of text a model reads and bills by.

Where the full-source agent went wrong

Mode A's context was full of internal helper code, checksum loops and old comments about a superseded version of the rules. The author argues this noise caused attention dilution, meaning the model loses track of the few lines that matter. That is the author's explanation, not something the test proved directly.

The final code does show two mistakes in validator.py. Mode A copied unvalidated extra keys from the input into its output, when the contract called for a strict five-key result. It also rejected a unit price of zero by writing cents <= 0 where the contract needed cents < 0.

Those slips cost it three of the 20 tests, for a score of 17 / 20 (85.0%). Mode B passed all 20 on its first turn.

Cost and speed

The efficiency gap was large, partly because Mode A needed three turns and the input grew each time. Its turns used 1,643, 14,712 and 28,915 input tokens. Mode B used 2,316 in a single call.

• Cumulative input tokens: 45,270 versus 2,316.
• Estimated cost: $0.0446 versus $0.0078, or -82.6%.
• Elapsed time: 29.99 sec versus 12.84 sec, or -57.2%.
• Tool output: 78,719 characters versus 105.

Mode B did write more output, 1,611 tokens against 1,249 on Mode A's second turn. But since it finished in one call, its total output was still lower, 1,611 against 2,839.

How far to trust it

This is one experiment on one task with one model, and the results come from the author's own harness. The post ranks the pattern #2 of 41 in the author's series. The full-source context was also partly synthetic: it included 12 generated internal helper modules built to act as noise.

Even so, the idea is practical. The post includes a short Python snippet that uses the standard ast module to fold function bodies into three dots while keeping signatures and docstrings. For projects in several languages, it suggests Tree-sitter.