AI gateway benchmarks reveal major cost and performance gaps

The benchmark setup

Researchers conducted a 24-hour benchmark to compare four major AI gateways: OpenRouter, Vercel, Ramp, and Keln. The test used the DeepSeek V4.1 Flash model to see how different infrastructure layers handle real-world agent sessions. Every gateway was configured with maximum reasoning and tool-calling settings enabled.

The test involved 13,440 total turns across the four gateways, with each gateway processing 3,360 turns. To mimic human behavior, 1 in 20 turns included a pause of one to 20 minutes. This setup allowed the team to observe how each gateway manages long-running conversations and repeated prompts over time.

The data was gathered between October 20 and October 21, 2026. The goal was to identify how different routing logic affects cache hit rates, speed, and total costs. As keln.ai reported, the results showed that even when using the same underlying model, the gateway's internal rules create vastly different user experiences.

Cache hit rate disparities

Cache hit rates varied significantly, ranging from 63% to 94% depending on the gateway used. Keln led the group with a 94% hit rate, while Ramp trailed at 63%. The primary reason for this gap is how each gateway selects a provider for subsequent turns in a conversation.

Keln maintains a conversation on the specific provider that holds its prefix. In contrast, Vercel pins a conversation to its first provider for only five minutes. OpenRouter and Ramp pick a provider again on every single request, which often leads to 'cold' prompts where the system must re-read tens of thousands of tokens at full price.

The impact of these choices is most visible after a 10-minute break. After such a pause, Keln still served 75% of the next prompt from the cache. During the same timeframe, Vercel and Ramp served only 25% and 23% of the prompt from the cache, respectively. This means users on Vercel or Ramp may face higher costs and slower response times for long-running tasks.

Reasoning and cost analysis

The benchmark also highlighted a major discrepancy in how gateways handle 'reasoning' tokens, which are the internal thoughts a model generates before answering. Every turn requested maximum reasoning, but Vercel and Ramp delivered less than half the reasoning volume that Keln and OpenRouter provided.

Because Vercel and Ramp returned fewer reasoning tokens without warning or error, their total bills were lower. Vercel's total cost for the run was $9.14, compared to $10.98 for OpenRouter and $11.26 for Keln. However, the report notes that if Vercel had provided the same volume of reasoning as Keln, its cost would have risen to $10.97.

Ramp's costs were the highest at $20.20, largely due to its poor cache performance. Ramp paid for 47.5 million uncached prompt tokens, while Keln only paid for 8 million. This demonstrates that a gateway's ability to persist context can be the difference between a manageable bill and a much higher one.

Provider switching behavior

A key metric in the study was how often a gateway moves a conversation to a different provider. Each move constitutes a 'cold prompt,' requiring the system to re-process the entire conversation history. This happens because the new provider does not have the previous turns stored in its local cache.

The data showed that Ramp moved conversations between providers 500 times during the test. OpenRouter moved them 148 times, and Vercel moved them 60 times. Keln was the most stable, moving conversations only 14 times. These frequent moves on the Ramp gateway contributed significantly to its higher costs and lower cache efficiency.

The study concludes that when the same provider behaves differently behind different gateways, the issue lies with the gateway's logic. For developers building AI agents, these findings suggest that the choice of gateway is just as important as the choice of model when it comes to balancing performance, reliability, and budget.

What this means for users

For developers and businesses, these results provide a clear picture of the hidden costs of AI infrastructure. A gateway that prioritizes provider variety or simple routing might save on immediate per-token costs but can lead to massive overspending on long conversations due to cache misses.

The benchmark shows that Keln's approach of keeping conversations on a single provider for the duration of a prefix provides the best cache hit rate. This stability ensures that the AI can 'remember' the context more efficiently, leading to lower costs and more consistent reasoning depth for complex tasks.

Moving forward, users should look closely at how their chosen gateway handles provider pinning and cache persistence. Understanding these underlying mechanics is essential for anyone trying to build scalable AI agents that remain cost-effective as conversation lengths grow over time.