VindexLLM runs local AI models on any Windows GPU, written in Delphi

What VindexLLM is

As github.com reported, VindexLLM is an engine for running language models on your own graphics card, written entirely in Delphi, a long-running programming language best known for Windows desktop software. It reads GGUF files, a common single-file format for local models, and does the whole job of turning a prompt into text on the GPU.

The project says the only thing it needs is vulkan-1.dll, a file that comes with the graphics drivers from NVIDIA, AMD and Intel. There is no Python environment, no CUDA toolkit (Nvidia's large developer package) and no separate runtime to install.

The heavy lifting is done by 37 small GPU programs called compute shaders. They are compiled ahead of time and baked into the executable, so the finished program is one self-contained file.

How it gets a model onto the GPU

The engine maps the model file into memory and copies the weights to the GPU's own memory (VRAM) through staging buffers. It does not convert the file into some custom format first. Once loaded, the work for each token, including attention and feed-forward steps, stays on the GPU.

The project says only two things cross the connection between the computer and the card for each token. One is a tiny input, about 10KB, for the token's embedding. The other is the final scores for the next word, which are sent back so the program can choose one.

The tokenizer, which splits text into pieces the model understands, is also written in Delphi and reads its vocabulary straight from the GGUF file. Sampling controls include temperature, top-K, top-P, min-P and a repetition penalty, plus a fixed random seed for repeatable output.

Squeezing the memory a chat needs

As a model generates text, it keeps notes on everything it has already read. This store is called the KV cache, and it grows with the length of the conversation. VindexLLM includes its own compression scheme for it, named TurboQuant or TQ3.

It works on blocks of 32 numbers. First it scrambles them with a mathematical transform (Walsh-Hadamard) so information is spread evenly, then it rounds each to one of eight levels, which is 3 bits. The project claims about 9× less memory than the uncompressed F32 format and says quality beats naive 3-bit rounding. The attention step can read the compressed data directly.

Those are the author's claims, and what we have includes no independent quality tests.

Memory and search built in

Beyond plain text generation, the repo includes a console chat program. It stores past conversation turns in a SQLite database and can search them later, using a second small model, EmbeddingGemma 300M, to find relevant older turns. This technique, known as RAG (retrieval-augmented generation), feeds the matching snippets back to the main model as reference.

The author says the full stack, with chat model, embedding model and memory, should fit comfortably on most consumer GPUs with 4–6 GB of VRAM. That assumes the 4-bit versions the project vetted, which were trained with quantization in mind so they lose less quality.

Limits and what to check

The big limit is model support. Only the Gemma 3 4B architecture works today, and other models are rejected with an error message. Adding another family means writing its specific layer structure.

Speed is less clear. The repo says it measured Gemma 3 4B in Q8_0 on an NVIDIA RTX 3060 12GB, but the numbers are not in the text we reviewed, so we cannot say how it compares with established tools.

The code is under the Apache 2.0 license, which allows commercial use and modification, as long as you keep the copyright notice and license file. To build it, you need Delphi 12 or higher and a Win64 target, and the compiled shaders are already in the repo.