Most LLM bills are inflated by sending every request to the priciest model
A dev.to guide says teams can cut AI costs by tracking spend, caching repeated text and routing easy tasks to cheaper models, without losing quality.
The short version
- The author says overspending comes from treating all requests alike, not from inefficient models.
- Step one is logging tokens and cost per feature, so you know where the money goes.
- Prompt caching is said to save 40-80% on long, repeated instructions or documents.
- Model routing is called the biggest lever, with claimed savings of 40-70%.
- Provider batch APIs for non-urgent jobs typically give a 50% discount, per the post.
Why it matters
If your team pays per use for an AI feature, the bill is shaped by design choices you control, not just by vendor prices. The post's advice is cheap to try, and the first step is simply measuring. Treat the savings figures as the author's claims until your own numbers confirm them.
The problem: every request treated alike
As dev.to reported, in a post by Tanmay, who builds AI agents and automation for businesses, API bills for AI features can grow startlingly fast once they reach production. His diagnosis is that the models are not wasteful. The wasteful part is that every request, from a trivial label-this-message question to a multi-step reasoning task, tends to go to the same top-tier model.
The title promises three strategies, but the post lays out four steps. The first is groundwork (measurement), and the last bundles two smaller tweaks. The order matters, because each step builds on the one before.
Step one: find out where the money goes
The author's first rule is that you cannot cut what you do not measure. Before touching prompts or swapping models, he suggests recording how many tokens go in and out of each call, and what that costs, tagged by feature. A token is the small chunk of text, roughly a word fragment, that AI providers use to bill you.
His example is a short Python wrapper that multiplies token counts by a price table and writes a log line naming the feature and model. The prices in his example are illustrative: for a model he lists as gpt-4o, $2.50 per million input tokens and $10.00 per million output tokens, and far lower for the smaller gpt-4o-mini. The payoff is simple: you quickly see which part of your product eats the budget.
Caching: stop paying to re-read the same text
Many AI products, especially agents and systems that pull in documents to answer questions (known as retrieval-augmented generation, or RAG), send the same long instructions or reference material with every query. Re-processing identical text each time is wasteful. The post says modern providers offer prompt caching, where re-reading text they have already seen costs a fraction of the normal input price.
The author claims this often saves 40-80% on long system prompts or static document context. To benefit, he says, put the unchanging material first and the changing material, such as the user's question and the conversation so far, at the very end. Caching works on the repeated start of a prompt, so the order matters.
Routing: match the model to the task
The post calls model routing usually the single biggest lever, with claimed cuts of 40-70%. The idea is that jobs like sorting a request into a category, pulling out structured data, or spotting names in text do not need a flagship model. A smaller, cheaper one can handle them.
His pattern is a two-stage setup. A cheap model first reads the incoming question and labels it simple or complex. Simple ones are answered by the cheap model, and only complex ones go to the expensive one. The classifier step costs a little, but a large share of routine traffic never reaches the pricey model.
One detail worth noting: his routing code logs its own classifier calls under a separate feature name. That means the tracking from step one also shows whether the router itself is worth its cost.
Trim the prompt, batch the slow work
Once the first three are in place, the author suggests two finishing moves. Prompt compression means cutting wordy instructions, dropping old messages from a conversation's memory, and removing formatting that adds tokens without adding meaning.
Batching applies to work nobody is waiting on, such as nightly document indexing or running evaluations. Providers' batch interfaces, he says, typically charge 50% less per token for this kind of job, in exchange for not answering instantly.
A caution: the post gives percentages but no measurements from a real system, and it says output quality stays intact without showing a test. The savings you see will depend on your traffic. If most of your requests are genuinely hard, routing will help less. If your prompts are short, caching will help less.
What to watch
- Do the claimed savings of 40-80% for caching and 40-70% for routing hold up in real products, and does answer quality stay the same?
- How much does the extra classifier step in routing add in cost and delay for products with mostly hard requests?
Try it
Pick one AI-powered feature you run or pay for. Write down its monthly cost, how many calls it makes, and how much of each prompt is identical text repeated every time.
Sources and links
- Reportingdev.to: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by dev.to · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
