vLLM says its open engine is within 6% of DeepSeek's in-house DeepInfer

What vLLM actually claimed

On October 24 (UTC), the vLLM project announced that it had matched the performance curve of DeepInfer, the inference engine DeepSeek open-sourced on October 20 under the Apache 2.0 license. As BAAI Community reported, vLLM says it got there by building in DeepSeek's open-source kernels (low-level routines that do the heavy GPU math) and tuning both stages of answering a prompt.

The match covers DeepSeek's own decode configuration for its V4.1-Flash model, across batch sizes from 8 to 256. Batch size is how many requests the system handles at once. An accompanying engineering blog puts the gap at within 6%.

It is a claim about a whole curve, not one favorable data point. That is harder to achieve than matching a single number.

How the comparison was built

The details matter here. In the blog's setup, the context length is 131,072 tokens and a setting called nextn=3 is used. The vLLM numbers come from GB200 hardware, and the measure is output throughput per GPU as batch size changes.

vLLM did not run DeepInfer itself. The blog says the DeepInfer numbers were read from charts in its README. Those charts were measured on 64 NVIDIA SM100 GPUs in a setup labelled EP64DP64, while vLLM used a 32-GPU configuration called DEP32, which combines data-parallel attention with expert parallelism. Expert parallelism spreads a model's specialist sub-networks across several chips.

So this is vLLM's own measurements set against another team's published figures, not a head-to-head run on identical machines.

Why prefill gets a weaker verdict

Prefill is the stage where the model reads your whole prompt before it starts writing. Here the blog is more cautious. DeepInfer's prefill chart does not say what hardware it used, and its settings differ from vLLM's in several ways, including the parallel layout, the number of experts and the replay window.

Because of those differences, the blog declines to call the gap a speedup of one engine over the other. It says only that vLLM's smaller prefill setup follows the same trend inside DeepInfer's published range.

That restraint is useful. It tells you which part of the claim the authors themselves consider solid.

Why the two engines are close relatives

The blog explains the similarity as shared lineage. DeepInfer was originally built mostly on vLLM, with some ideas from SGLang and TensorRT-LLM, two other serving engines. In return, vLLM keeps absorbing kernels DeepSeek releases, and the blog names MegaAttn, MegaMoE and Mega-mHC. It then tuned the rest of its stack around them.

The blog sums up the lesson with one line: a general engine can match a specialized one. That fits what the numbers show for offline decoding, but it is vLLM's own conclusion.

The work was led by the Inferact team. vLLM thanked DeepSeek, NVIDIA and SemiAnalysis.

Online results and the missing pieces

The same thread reported online-serving numbers from vLLM's own build, not DeepInfer's. On one GB300 NVL72 rack, using SemiAnalysis's AgentX benchmark, vLLM reached 954K tokens per second per GPU. That counts cached input, uncached input and output together. It also reported 196 tokens per second per user at the 90th percentile, and an equivalent cost of $38.96 per GPU hour under SemiAnalysis's cost model. The throughput is 7.9 times vLLM's own single-machine result from October 7.

Other figures: a dedicated low-latency decode path hit 826 tokens per second for one user on 4 B300 GPUs, against 575 for the standard path. Fixing one data-transfer step (NIXL push) lifted per-GPU throughput from 141K to 440K tokens per second.

DeepInfer is a different kind of release. It supports only DeepSeek-V4.1-Flash, offers offline inference only, and its README lists the KV cache system, prefill/decode separation and load balancing as not included. The KV cache is the memory of earlier tokens. On October 21, SemiAnalysis argued these gaps made online comparisons impossible. vLLM's result speaks to the offline half; the online half is still open for DeepInfer itself.

How much to trust it

Treat it as a credible but unverified claim. Technology.org reported on October 21 that no outside group had tested DeepInfer's numbers, and vLLM's comparison relies on those same published charts. Its own 954K and $38.96 figures are also self-reported and have not been reproduced by a third party.

The 6% figure holds for one model, one hardware setup and one context length. Your traffic, prompt lengths and latency needs may behave differently.