SIAgentNews.com
← All news

How to prove a self-hosted AI model is ready for agent work before you switch

A new arXiv paper offers a testing pipeline for self-hosted models running agents, and finds that public benchmark traces can badly overstate how much load a system handles.

· First reported by arxiv.org

1.18xReplay vs recorded input tokens, 95th percentile

The short version

  • Companies are moving agent workloads from closed APIs to open-weight models they host themselves.
  • Replaying recorded agent sessions stayed within 1.18x of recorded input tokens at the 95th percentile; live re-runs reached 4.98x.
  • Live, replayed and synthetic workloads each exercised serving code the others missed in vLLM.
  • In a four-model rollout on llm-d, captured workloads reached about half the optimal concurrency and about 60% of the throughput seen with public traces.
  • The authors open-source their tools and release a dataset of traces.

Why it matters

If you are thinking about moving agents from a paid API to a model you host, a demo that works once tells you little. This paper argues you should test with recordings of your own agents' work, because borrowed or public test data can hide how the system really behaves under load.

The problem: testing by feel

Many companies are shifting their AI agents away from closed, pay-per-call services and onto open-weight models they run themselves. (Open-weight means the trained model files are downloadable, so you can host them on your own hardware.) As arxiv.org shows in a paper submitted on 19 Nov 2026, that shift makes testing much more important, and the authors say testing is often done badly.

Their complaint is specific. Rollouts are usually judged with ad hoc tools, and nobody checks whether those tools are accurate. In other words, the ruler used to decide whether a system is ready may itself be off.

The paper proposes an end-to-end evaluation pipeline for serving models to agents. It covers every stage: small-scale testing, sizing the hardware, pre-production, and production. The authors also say they tested the design choices that link those stages instead of just asserting them.

Replay beats re-running

An agent is a program that calls a model repeatedly to work through a task. Each call sends and receives tokens, the small chunks of text a model reads and writes. How many tokens each session uses largely determines how much load a server faces.

One way to test is to capture real agent sessions as traces, meaning saved records of what was sent. You can then replay them against a new setup. The other way is to run the agent live again. The paper compares the two against the original sessions.

Replay stayed close. At the 95th percentile, meaning the heavier end of sessions, replayed input tokens were within 1.18x of what was recorded. Live re-runs reached 4.98x. The authors conclude that a small set of recorded sessions can give a repeatable picture of per-session load. Live agents, by contrast, wander differently each time, which makes the load harder to compare from one test to the next.

Different test types reach different code

The researchers also looked at vLLM, a widely used piece of software for serving models. They ran a code-coverage analysis, which records which parts of a program's code actually executed during a test.

The finding is that live, replayed and synthetic (made-up) workloads complement each other in the configuration they measured. Each one exercised Python serving paths in vLLM that the others did not reach in the measured runs.

The practical reading is that no single kind of test covers everything. A mix of test types is more likely to expose problems than leaning on one, though the paper frames this result as specific to the setup it examined.

A case study with four large models

The paper then applies the approach to a rollout of four large mixture-of-experts models on llm-d. A mixture-of-experts model splits its work across specialist sub-networks, so only some of them run for any given request. The source does not describe llm-d beyond naming it as the platform used for the rollout.

The comparison was between public traces and the team's own captured workloads. On captured workloads, the target models reached about half the optimal concurrency, which is how many requests the system can handle at once. Correspondingly, they reached about 60% of the throughput, the amount of work finished over time, compared with public traces.

That gap matters for planning. If you size hardware using public benchmark traffic, you may expect noticeably more than your own agents will get. The authors say they are open-sourcing the relevant tools and releasing a trace dataset, so others can try the approach.

What to take from it

The paper is not a product launch, and the abstract does not say how the pipeline performs on every model or setup. The results come from the configurations the authors tested, so treat them as strong evidence for a habit rather than a universal law.

The habit is simple: record real agent sessions, replay them when you change models or hosting, and compare outcomes and speed under that load. A one-off demo, or a benchmark built on someone else's traffic, can leave you with a system that looks ready and then falls short.

What to watch

  • Whether other teams reproduce the replay results using the released tools and trace dataset.
  • Whether the gap between public traces and captured workloads holds for other models and serving platforms.

Try it

Before switching models or hosting, save a handful of real agent sessions and replay them against the new setup, comparing success and speed with your current one.

Sources and links

First reported by arxiv.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news