SI Agent News
DevelopingOpen modelsResearch

A shrunken 9B model ran 52% faster with no measured quality loss in one test

A hobbyist's benchmark of a community-made model found that dropping to a smaller file format cost nothing measurable, which matters for anyone running AI at home.

1.5xGeneration speed gain, Q4_K_M vs Q8_0

The short version

Why it matters

If you run models on your own computer, file size and speed decide what fits and how pleasant it feels. This test suggests the smaller format gave up nothing noticeable for chat on a 9B-class model. It is one person's measurement, so treat it as a strong hint to check yourself rather than a rule.

What the author measured

A writer who goes by Sactasura posted a measurement report on note about a model called DeepSeek-V4-Pro-Qwen3.5-9B. It is a distilled model made by the community, not an official DeepSeek release. (Distilling means training a smaller model to imitate a bigger one.) The question was simple: how much worse does the model get when you store it in a more compressed format?

He compared two versions of the same model, Q8_0 and Q4_K_M, on the same evaluation set. The smaller Q4_K_M version generated text at 97.9 tokens per second, against 64.5 for Q8_0. That is about 1.5 times faster, or +52%. The file shrank from 9.10 GiB to 5.37 GiB.

His conclusion was that the speed and size gains came almost free, because he could not measure any loss in quality. As note's post lays out, this is a practical verdict, not a lab-grade study.

What quantization actually does

A model is a huge collection of numbers. Quantization stores those numbers with fewer bits of precision, a bit like saving a photo at lower resolution. The file gets smaller and the computer has less data to move around, so it usually runs faster.

Q8_0 and Q4_K_M are labels used in the llama.cpp ecosystem, a popular open-source program for running models on personal machines. Roughly, the number in the name tells you how many bits are kept per value, so Q4 is the more compressed of the two.

The usual fear is that squeezing too hard makes the model forgetful or sloppy. This test is an attempt to see whether that happens at this particular step down.

The test setup and the scores

The machine was a desktop with a Ryzen 7 9800X3D processor, 64GB of memory and an RX 9070 XT graphics card with 16GB. He used the Vulkan version of llama.cpp, loaded every layer onto the graphics card, set an 8192-token context window and a temperature of 0, and capped answers at 800 tokens. Temperature 0 makes the model pick its most likely word each time, which keeps repeated runs comparable.

The evaluation set was his own: 10 Japanese questions, 30 refusal tests, 8 adult-oriented creative writing tasks, plus extra conversation checks. Time to first token, the wait before the answer starts appearing, was 0.118 seconds.

One limit is worth knowing: the Q8_0 and heretic numbers were re-posted from his earlier measurements, not re-run this time.

Where the model still stumbles

The mistakes the model made were the same at both precision levels. It confidently said the Kanto region has 13 prefectures, when the correct answer is one metropolis and six prefectures. It also gave 5 as the number of sounds in a haiku, where the correct answer is 17.

Because Q4 and Q8 both got these wrong, the author argues that compression did not remove the knowledge. The base model simply had it wrong. Its polite emails also came out as templates full of placeholders such as month and day, so he suggests treating them as drafts.

The uncensored heretic version had one failure the others did not: in an email task it looped, repeating a stock phrase more than 30 times until it hit the length limit. Refusals were zero across all three versions, which he says is expected for a base model.

How far to trust it

This is one author's test on one model, using a small, self-built question set that is mostly in Japanese. A set that size can easily miss narrow losses, such as long reasoning or code. The author himself says only that degradation could not be measured in this set, which is weaker than saying none exists.

Still, the details are unusually open: the hardware, software settings and test categories are all stated, and he offers a full report file for anyone who wants to repeat it. His own advice is that Q4_K_M looks like the sensible choice for 9B-class models, and that fact-checking remains a must because the model states errors with confidence.

What to watch

Try it

Download both the Q8_0 and Q4_K_M files of a model you use, run 20 of your own real prompts through each, and compare. If you can't tell the answers apart, keep the smaller file.

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by note · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news