OasisKV moves AI memory off the GPU and fetches only what the model needs next

The memory squeeze behind long chats

As arxiv.org reported, the paper starts from a simple observation: serving large language models is increasingly limited by memory rather than computing power. When a model writes a reply, it keeps a running record of the text so far, called the KV cache (key-value cache). With long prompts and long step-by-step reasoning, that record becomes the biggest user of memory.

That record normally sits in HBM, high-bandwidth memory, the fast and costly memory attached to the GPU. When it fills up, the number of requests a server can handle at once shrinks, and so does overall throughput.

Keep a little close, fetch the rest

OasisKV pushes the cache out to other kinds of memory, and brings back only the pieces that matter. The authors rely on the fact that, at each step, the model pays close attention to only a small share of earlier text. They also use speculative decoding, a technique where the system guesses upcoming words ahead of time, to predict which earlier pieces will be important.

An asynchronous prefetch pipeline (fetching in the background while the GPU works) hides the delay of moving data from outside the GPU. It also caps how much data moves per step so it fits the available bandwidth.

What the authors measured

The team built OasisKV on vLLM, a widely used serving system. With only 2,048 KV tokens held in HBM, average accuracy stayed within 0.61 points of full attention on LongBench v2, a long-document test, and on AIME24-25 and GPQA-Diamond, which test long-form reasoning.

On AIME24, throughput rose 1.86x over vLLM. Synthetic long-context workloads on multiple GPUs showed up to 2.2x higher throughput.

The system also manages the cache across decode, prefill and remote-memory nodes. In a setup that splits those stages across machines, OasisKV reached 2.29x vLLM's throughput and supported 2.42x as many concurrent requests on the same host-memory budget.

Caveats

These are the authors' own results, and some come from synthetic workloads rather than real traffic. This is an updated version of the paper on arXiv. Whether the approach lands in everyday tools will depend on how easily others can adopt it.