Supermicro, IBM and NVIDIA launch AI caching to cut response times

Solving the prefill bottleneck

When a user asks an AI to analyze a long document, the system must first perform a stage called prefill. During this phase, the model converts the input into tokens and performs calculations to understand the context before it can begin generating an answer.

The problem is that without a way to remember these calculations, the system may have to re-process the same information repeatedly. This is especially costly when an AI assistant needs to refer to the same large report while answering several different follow-up questions from a user.

The new infrastructure from Supermicro, IBM, and NVIDIA addresses this by using a method called key-value (KV) caching. This technology stores the intermediate information from these initial calculations so the model can reuse them instead of performing the same work every time a new request arrives.

Moving beyond GPU memory

Standard AI systems usually keep frequently accessed information within the memory of a graphics processing unit (GPU). However, this memory is both expensive and limited in capacity, which creates a bottleneck for applications involving large software code, images, or extended conversations.

When the data exceeds what a single GPU can hold, systems often have to discard the cached data or move it to slower storage. The new solution allows previously calculated information to be stored outside of the GPU and retrieved only when it is required by the model.

By creating a storage layer that operates beyond individual GPU limits, the companies aim to allow multiple AI systems to access shared information. This approach helps reduce unnecessary calculations while allowing data centers to serve more requests without needing to increase computing resources at the same rate.

Benchmark performance results

The companies tested the system using a large language model with 120 billion parameters, which are the internal values an AI model learns during training. In one test involving a prompt of approximately 130,000 tokens, the system delivered a 56-fold improvement in time-to-first-token, which is the delay before the AI begins its response.

Specifically, processing the long prompt without the shared caching system took about 32.14 seconds. With the cache enabled, that time dropped to approximately 0.57 seconds. The companies also noted that as prompt length increased from 30,000 to 130,000 tokens, the delay with caching only grew by 0.377 seconds, while it grew by 29.55 seconds without it.

The system also showed a significant boost in throughput, which measures how many requests can be handled simultaneously. Under benchmark conditions, throughput increased from 0.19 to 4.26 requests per second, representing an approximately 22-fold improvement in the system's ability to handle multiple tasks at once.

Hardware and software integration

The platform is a combination of specialized technologies from three different infrastructure leaders. Supermicro provides Petascale storage servers powered by AMD EPYC 9005-series processors to handle the heavy lifting of data storage.

IBM Storage Scale software manages the shared storage, allowing different computing systems to retrieve the cached AI information. Meanwhile, NVIDIA provides the Dynamo inference framework to coordinate distributed AI inference and Spectrum-X Ethernet networking to support communication between the servers.

This integrated stack is designed to support high-demand applications like coding assistants, document analysis, and retrieval-augmented generation. Retrieval-augmented generation is a process where AI systems draw on external databases or documents to prepare more accurate and context-aware responses for the user.

Commercial and operational impact

As AI infrastructure grows more expensive, businesses are facing constraints regarding electricity, cooling, and hardware availability. This new technology highlights a shift toward reducing the costs of operating models after they have already been trained, rather than just focusing on the initial training phase.

The companies claim these improvements could help businesses make better use of their existing GPUs by increasing efficiency. However, the announcement does not guarantee a universal reduction in electricity or operating costs, as those benefits depend on specific workloads and how often cached information can be reused.

While the 56-fold improvement is a notable benchmark, the actual commercial impact will depend on whether these results can be reproduced in diverse environments. Supermicro is currently offering integration and deployment services to help businesses install the system, though the companies have not yet named specific production customers.