Friendli says its engine serves 2.1× more tokens than vLLM on the same GPUs

What Friendli claims

Friendli, a company that sells an inference engine (the software that runs an AI model and produces its answers), published a comparison against vLLM, a widely used open-source engine for the same job. As friendli.ai reported on its own blog on October 29, 2026, both engines ran on four NVIDIA B200 GPUs, using an agentic workload built around a model called GLM-5.3-Flash.

The post makes two separate claims. At a fixed per-user speed, Friendli says it produces about 2.1 times as many tokens per second overall. In a second comparison, where total output is held equal to vLLM's best, it says each user gets text about 3.1 times as fast.

A token is a small chunk of text, roughly a word or part of one. It is the unit these systems count in.

Speed and capacity pull apart

The post's main teaching point is that two numbers both measured in tokens per second describe different things. Per-request generation speed is how quickly one answer streams out. Aggregate throughput is the total output across everyone using the system at once.

These can move in opposite directions. The post gives a simple example: ten responses at 100 tokens per second each add up to 1,000 tokens per second. Twenty responses at 75 each add up to 1,500. Total output rose 50%, yet every individual response got 25% slower.

So a bare throughput figure can hide a worse experience for each user. Friendli's answer is to plot the two measures together, with one point for each operating condition. The result is a curve showing which combinations of speed and capacity an engine can deliver.

Two ways to read the curve

The first reading fixes the speed. At 100 output tokens per second per request, Friendli reports about 2.1 times vLLM's total throughput. Because the same GPUs finish the same amount of work in less time, the post says that implies about 53% lower GPU cost per output token, assuming equal GPU-hour prices and a deployment that keeps those rates up.

The second reading fixes the capacity. At vLLM's measured peak throughput, the post puts vLLM at about 81 tokens per second per request and Friendli at 249. For a 500-token reply, that is roughly 6.2 seconds against 2.0.

The post ties this to agents, which are programs that take multi-step actions. An agent often has to finish writing a tool call before it can run it, so a shorter wait lets the next step start sooner. Across many sequential calls the savings can add up. The post adds that the real benefit depends on output lengths and on how much of a task's time goes to generating text.

Why a grain of salt is warranted

This is a vendor comparing its own product with a rival, using its own chosen workload. That does not make the numbers wrong, but it does mean they show the best case for one side. The text we reviewed does not describe how vLLM was configured, and the results come from a single model on a single hardware setup.

The post itself points in the same direction. It tells readers to use their own model, typical input and output lengths, and expected traffic to find the operating point that fits their workload. It then invites readers to talk to a Friendli engineer to benchmark their own case.

The most durable idea here is not the 2.1× figure. It is the habit of asking for a requirement first, such as a minimum speed per user, and then comparing what each engine delivers at that requirement.

What to do with it

If you serve open models, treat this as a template for your own test. Pick the response speed your users or agents actually need, then see how much total traffic each engine handles at that speed on your hardware.

Do the reverse as well: fix the traffic you expect and see how fast individual answers arrive. A vendor curve cannot tell you where your own workload sits on it.

• Use your real prompts and typical reply lengths.
• Match the number of simultaneous users you expect.
• Record both per-user speed and total throughput, not just a peak.