llama.cpp build b12533 fixes a CUDA crash on very large output matrices
A small fix to Nvidia GPU code stops a number overflow that crashed huge-output runs, a reminder of how easily big models can break old assumptions.
The short version
- The old code stored output-tile positions in a 32-bit integer, which wraps once outputs pass 2^31 elements.
- The failing example: a 248320 x 16384 output, about 4.07e9 elements, on a q8_0 output layer.
- Symptom: an illegal memory access that aborts the program, seen with llama-perplexity at large batch sizes.
- The fix widens the offsets to int64_t in mul_mat_q and the stream-k fixup step.
- A new test is included but is off by default because it needs about 9 GB.
Why it matters
Models with very large vocabularies, run with large batches, can push GPU code past limits that were once safe. This fix removes one such limit for Nvidia users of llama.cpp. Most people will never hit it, but anyone running big evaluations should update.
What broke
As github.com reported in the release notes for build b12533, the GPU code that multiplies compressed (quantized) model weights computed an output position using a 32-bit integer. That type tops out around two billion, or 2^31.
When the output matrix had more elements than that, the position math wrapped around to a wrong value. The program then tried to write outside its memory, and the GPU reported an illegal memory access and stopped.
The case that exposed it
The notes give a concrete trigger. Running llama-perplexity, a tool that scores how well a model predicts text, with a context of 32768 and a physical batch of 16384, on a model with a 248320-word vocabulary.
That setup keeps the model's predictions for every position, so the final q8_0 output layer produces a 248320 x 16384 result. That is about 4.07e9 elements, well past the 32-bit limit.
The fix and the safety net
The change widens the offsets to 64-bit integers in two places: the main matrix routine, mul_mat_q, and the stream-k fixup, a step that reconciles partial results when work is split across the GPU.
It also adds a regression test to the project's backend test suite. It is disabled by default because it needs roughly 9 GB. The commit also credits an AI assistant, Claude Opus 5.5.
Who should care
The described trigger is narrow: Nvidia GPUs, very large outputs, and big batch settings. Everyday chat use is not what the notes describe. People who evaluate models at long context or build large-batch pipelines are the likelier audience.
What to watch
- Whether other GPU backends or code paths hold similar 32-bit limits as vocabularies and batches grow.
- Whether the disabled-by-default test ever gets run in regular checks, given its 9 GB cost.
Try it
If you run llama-perplexity or similar large-batch jobs on an Nvidia GPU and hit an illegal memory access, update to b12533 or later and rerun the same command.
Sources and links
- Reportinggithub.com: the article we sourced
First reported by github.com. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.