Cheap AI models compared: DeepSeek Flash and Claude Haiku tie on accuracy, not on cost
A 17-task test found both budget models passed 50 of 51 runs, but their bills, speed and price changes differ in ways that matter at high volume.
The short version
- Both models passed 50 of 51 runs across coding, extraction and reasoning tasks.
- Median latency: Haiku 2.8 s, Flash 9.9 s with default thinking, 3.6 s with reasoning off.
- Flash averaged 1,730 output tokens per task, 1,518 of them hidden reasoning that is billed.
- Cost per 1,000 tasks: Flash $0.97 at test-day prices, $0.40 after its cut; Haiku $0.13.
- The sample is small and short-prompt only; long documents and tool calls were not tested.
Why it matters
Cheap models handle most real volume, so small differences in how they bill add up quickly. This test shows that price per token is a poor guide: what counts is the cost per finished task, which depends on how much a model writes, including hidden thinking. It also gives you a method you can copy with your own work.
What the test compared
As dev.to reported, in a post first published on the QuickSilver Pro blog, the authors pitted two budget models against each other: DeepSeek V4.1 Flash and Claude Haiku 5.5. These are the models people pick for jobs that repeat thousands of times a day, such as sorting messages into categories, pulling fields out of text, making small code edits and handling the background calls of an AI agent.
The question was not which model is cheaper per token, since Haiku is on both input and output. It was which one is cheaper per finished task, and whether either gets answers wrong more often.
The run took place on 2026-10-20 through the QuickSilver Pro API, a service that offers both models behind one interface.
How the 17 tasks were built
Every task had an answer a script could verify, so no human judged quality. There were 7 Python coding tasks (a duration parser, interval merging, an LRU cache, strict Roman numerals, an expression evaluator, semantic-version ordering and a shell-style word splitter). Code passed only if it cleared every unit test.
Four extraction tasks asked for structured data, such as an invoice from an email or the final state of an order after a messy chat. The output was compared to the expected answer field by field. Six reasoning tasks had known numeric or one-word answers, including a pipes-and-tank problem and an elimination puzzle.
Each model got the same single message per task, with no system prompt and no temperature setting. Each task ran three times, giving 51 requests per model.
Same accuracy, very different bill
Both models passed 50 of 51 runs. The gap showed up elsewhere. Flash thinks before it answers by default, producing a hidden chain of reasoning that is billed as output. It averaged 1,730 output tokens per task, and 1,518 of those were reasoning. Haiku averaged 834.
- Speed: Haiku's median was 2.8 seconds against 9.9 for Flash. At the 90th percentile it was 7.2 seconds against 40.5.
- Cost per 1,000 tasks at test-day prices: $0.97 for Flash, $0.13 for Haiku, roughly a seventh.
- Cost after Flash's cut: DeepSeek lowered Flash's rates on 2026-11-06, to $0.0558 for input and $0.2231 for output per million tokens. The authors recomputed from the same measured tokens without rerunning the test, which brings Flash to $0.40 per 1,000 tasks, still about three times Haiku.
The misses differed in kind. Flash once gave a wrong number on the elimination puzzle. Haiku's single miss was an answer that stopped partway through a function. In both cases the other two runs passed.
Switching the thinking off
Flash can skip its reasoning step. On QuickSilver Pro that means sending a request setting that disables it, and the authors confirmed the replies then reported zero reasoning tokens. With it off, Flash averaged 556 output tokens and its median latency fell to 3.6 seconds.
The trade was a little accuracy. It passed 48 of 51, missing the elimination puzzle twice and the duration parser once. At test-day prices it was still about 2.5 times Haiku's cost. After the cut, both land at $0.13 per 1,000 tasks.
The authors caution that 50 versus 48 out of 51 means "about the same", not a ranking. One task more or less moves a pass rate by two points.
Where each model fits, and the limits
The authors lean toward Haiku for high-volume work with short prompts, since it matched Flash's accuracy at lower cost and answered faster. They suggest Flash when prompts are very long. Haiku's rates climb to $0.125 for input and $0.625 for output once a prompt exceeds 200K tokens, while Flash charges one rate across its 1M-token context. That long-prompt case was not tested, so it is a pricing observation rather than a result.
The test has real limits. It covers short, self-contained, English-language tasks. Long documents, multi-turn chats, tool calls and other languages were not tried. Latency came from one client on one day and shifts with load.
One pass of all 17 tasks on both models cost about two cents, so the experiment is cheap to repeat on your own material.
What to watch
- Does the gap hold on long prompts, multi-turn chats, tool calls and non-English text, none of which were tested?
- Will further price changes shift the cost ranking again, as DeepSeek's 2026-11-06 cut did?
Try it
Write 15 to 20 tasks from your own work with answers a script can check, run both models three times each, and compare cost per correct answer, not price per token.
Sources and links
- Reportingdev.to: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by dev.to · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.