A shrunken 9B model ran 52% faster with no measured quality loss in one test
A hobbyist's benchmark of a community-made model found that dropping to a smaller file format cost nothing measurable, which matters for anyone running AI at home.
The short version
- Speed rose from 64.5 to 97.9 tokens per second, about 1.5x (+52%)
- File size fell from 9.10 GiB to 5.37 GiB, roughly 40% smaller
- Japanese score was 33/50 against 32 for Q8_0; conversation scored 20/20
- Errors such as the Kanto prefecture count appeared in both versions, so compression did not cause them
- This is a community distill and one author's own test, not official DeepSeek
Why it matters
If you run models on your own computer, file size and speed decide what fits and how pleasant it feels. This test suggests the smaller format gave up nothing noticeable for chat on a 9B-class model. It is one person's measurement, so treat it as a strong hint to check yourself rather than a rule.
What the author measured
A writer who goes by Sactasura posted a measurement report on note about a model called DeepSeek-V4-Pro-Qwen3.5-9B. It is a distilled model made by the community, not an official DeepSeek release. (Distilling means training a smaller model to imitate a bigger one.) The question was simple: how much worse does the model get when you store it in a more compressed format?
He compared two versions of the same model, Q8_0 and Q4_K_M, on the same evaluation set. The smaller Q4_K_M version generated text at 97.9 tokens per second, against 64.5 for Q8_0. That is about 1.5 times faster, or +52%. The file shrank from 9.10 GiB to 5.37 GiB.
His conclusion was that the speed and size gains came almost free, because he could not measure any loss in quality. As note's post lays out, this is a practical verdict, not a lab-grade study.
What quantization actually does
A model is a huge collection of numbers. Quantization stores those numbers with fewer bits of precision, a bit like saving a photo at lower resolution. The file gets smaller and the computer has less data to move around, so it usually runs faster.
Q8_0 and Q4_K_M are labels used in the llama.cpp ecosystem, a popular open-source program for running models on personal machines. Roughly, the number in the name tells you how many bits are kept per value, so Q4 is the more compressed of the two.
The usual fear is that squeezing too hard makes the model forgetful or sloppy. This test is an attempt to see whether that happens at this particular step down.
The test setup and the scores
The machine was a desktop with a Ryzen 7 9800X3D processor, 64GB of memory and an RX 9070 XT graphics card with 16GB. He used the Vulkan version of llama.cpp, loaded every layer onto the graphics card, set an 8192-token context window and a temperature of 0, and capped answers at 800 tokens. Temperature 0 makes the model pick its most likely word each time, which keeps repeated runs comparable.
The evaluation set was his own: 10 Japanese questions, 30 refusal tests, 8 adult-oriented creative writing tasks, plus extra conversation checks. Time to first token, the wait before the answer starts appearing, was 0.118 seconds.
- Japanese: 33 out of 50 for Q4_K_M, against 32 for Q8_0. An uncensored variant called heretic, also at Q4_K_M, scored 34. He calls all three within the margin of error.
- Conversation: a perfect 20 out of 20, including remembering names, places and the names of two cats.
- Harder Japanese tasks: 21 out of 25. The lost points came from a summary that missed key figures.
One limit is worth knowing: the Q8_0 and heretic numbers were re-posted from his earlier measurements, not re-run this time.
Where the model still stumbles
The mistakes the model made were the same at both precision levels. It confidently said the Kanto region has 13 prefectures, when the correct answer is one metropolis and six prefectures. It also gave 5 as the number of sounds in a haiku, where the correct answer is 17.
Because Q4 and Q8 both got these wrong, the author argues that compression did not remove the knowledge. The base model simply had it wrong. Its polite emails also came out as templates full of placeholders such as month and day, so he suggests treating them as drafts.
The uncensored heretic version had one failure the others did not: in an email task it looped, repeating a stock phrase more than 30 times until it hit the length limit. Refusals were zero across all three versions, which he says is expected for a base model.
How far to trust it
This is one author's test on one model, using a small, self-built question set that is mostly in Japanese. A set that size can easily miss narrow losses, such as long reasoning or code. The author himself says only that degradation could not be measured in this set, which is weaker than saying none exists.
Still, the details are unusually open: the hardware, software settings and test categories are all stated, and he offers a full report file for anyone who wants to repeat it. His own advice is that Q4_K_M looks like the sensible choice for 9B-class models, and that fact-checking remains a must because the model states errors with confidence.
What to watch
- Whether other people repeat the Q8_0 to Q4_K_M comparison on larger models and on tasks like code and long reasoning
- Whether the loop failure seen in the uncensored variant shows up for other users
Try it
Download both the Q8_0 and Q4_K_M files of a model you use, run 20 of your own real prompts through each, and compare. If you can't tell the answers apart, keep the smaller file.
Sources and links
- Reportingnote: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by note · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
