Why picking the top few matching tools stops working when an agent has 1,000

Why the simple version runs out

As dev.to reported in an engineering post, the common recipe for choosing tools is to turn each tool description into a searchable fingerprint, fetch the closest matches, and let the language model pick one. The author says this holds up at about 100 tools. At 1,000 it needs what he calls a control plane: a set of checks around the model.

The arithmetic explains why. At roughly 400 tokens (small chunks of text the model reads) per tool definition, 100 tools cost 40k tokens. That is wasteful but it fits. At 1,000 tools the same math gives 400k tokens, so searching stops being a nice optimization and becomes the only way in.

Real tools can be heavier than that estimate. Audits the author cites put the 93 tools on GitHub's server at around 55,000 tokens in total.

Showing less helps the model

The post makes a point beyond saving space: a crowded context makes the model worse even when the right answer is in there. A well-known study, Lost in the Middle, found that accuracy dips when the relevant item sits mid-way through a long input. A long tool list has exactly that shape.

Several measurements point the same way, though they are different tests and should not be compared to each other. RAG-MCP reports 43.13% tool-selection accuracy with retrieval against 13.62% when the model is shown everything. Anthropic reports its Opus 4 model rising from 49% to 74% on an MCP test, and Opus 4.5 going from 79.5% to 88.1%, using about 85% fewer tokens.

The author's own figure is a simulation on a 120-tool catalog, where per-turn tool tokens fell from 47.3k to 2.4k (95.0%). He stresses that no live agent ran, so it shows savings, not accuracy. His design rule is to expose the smallest set of options that still contains the answer.

Tools are not categories

The post warns against treating tool choice as sorting items into labeled bins. A wrong bin is a bad row in a table. A wrong tool can mean a sent email, a deleted record or a charged card.

Other differences pile up. The list of valid tools changes per user and per session, because of permissions and live connections. Decisions come in sequences. Several tools from different servers often do nearly the same job. Even the tool descriptions are untrusted text, since a model tends to follow what a description says.

The conclusion is that safety cannot live in a prompt. It has to be code that runs whatever the model says.

Fix the catalog, then the search

In the simulation, 29 of 500 tasks failed even with tool gating. About 48% came from a request matching several similar tools, and 24% from cryptic legacy descriptions. So 72% of failures started with what the catalog said. The author calls this a hypothesis to test on your own logs, not a law.

His answer is to treat each tool as a product, with a card written for search: a summary phrased the way a user would ask, plus notes on when not to use it and which tool to prefer instead. Rewriting summaries in the user's voice cut their length by 63% and raised retrieval F1 (a combined score of finding the right items and avoiding wrong ones) by 8 points, on a synthetic benchmark.

Search then runs per sub-goal, since one request may hide two jobs. It combines three methods: meaning-based search, exact-word search (BM25) for ids and error codes, and matching on a structured list of operations. A merging method called reciprocal rank fusion, with its constant set to 60, blends the three rankings.

Filters, ranking and the recall chain

Permission checks should happen inside the search query, not after it. Otherwise the top 40 results can fill up with tools the user cannot call, and the measured recall still looks fine. Hard filters are plain rules: missing connection, wrong data type, a write tool for a read-only request, or anything policy forbids.

A reranker, a model trained to order candidates, then narrows the pool to a short menu. The author keeps the cut generous because a wrong cut deletes the right tool for good. He also disagrees with scoring permission and risk as penalties inside a weighted sum, since a slightly more relevant destructive tool could then outrank a safer one. He would make permission a filter.

His measurement advice is the most portable. End-to-end success is the product of each stage's conditional recall, meaning how often the right tool survives given it reached that stage. In his made-up illustration, the stages score 0.95, 0.98, 0.92 and 0.90, giving 0.771 overall. Improving the weakest stage often beats tuning the search everyone already tunes.