SIAgentNews.com
← All news

MIT Han Lab's Evolving Infra tunes model serving to agent workloads as they run

A new open-source serving system reshapes itself around coding, math and computer-use agents, and its authors report big latency gains over NVIDIA Dynamo.

· First reported by github.com

4.1×max cut in mean end-to-end latency vs NVIDIA Dynamo

The short version

  • Evolving Infra adapts to agent workloads online, with no offline profiling step beforehand.
  • Tested on up to 256 GB300 GPUs with GLM-5.2 and Kimi K2.6 serving coding, math and computer-use agents.
  • Versus NVIDIA Dynamo: up to 4.1× lower mean end-to-end latency and up to 14× lower P95 time to first token.
  • Throughput is up to 2.7× that of Dynamo, according to the authors.
  • The repository is built to be deployed and run by a coding agent, and ships a ready-made runtime image.

Why it matters

Agents make many linked requests that reuse long stretches of earlier context, which is a different pattern from one-off chat questions. If serving software is tuned to that pattern, the same GPUs can answer faster. Anyone paying for hosted inference or running their own models should expect agent-heavy use to behave differently from chat use.

What the Han Lab released

As github.com reported through the project's repository, MIT's Han Lab has published Evolving Infra, a system for serving large language models. Serving means the software that sits between users and the GPUs, deciding which machine handles which request.

The pitch is that it adapts itself to agent workloads purely while running, with no offline profiling, meaning no separate round of test runs to measure the workload first. The team tested it on coding, math and computer-use agents, using the models GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs.

Against NVIDIA Dynamo, another serving system, the authors report up to 4.1× lower mean end-to-end latency and up to 14× lower P95 time to first token. P95 means the slow end of responses: ninety-five out of a hundred arrive faster than this. They also report up to 2.7× the throughput. These are the authors' own numbers, and the benchmarks use math (Putnam), coding (Claude Code) and computer-use (OSWorld) agents at three concurrency levels each.

Why agents strain ordinary serving

Handling a request has two phases. Prefill is the model reading the prompt. Decode is the model writing its answer. Many serving systems give different machines to each phase, and the right split depends on the traffic.

Agents shift that balance constantly. They run long loops of tool calls, spawn sub-agents for side tasks, and compress their own history to stay within context limits. Each step reuses a lot of earlier text, which the system stores as a KV cache, a saved record of what the model has already processed so it does not redo the work.

A setup tuned for chat questions can end up with the wrong split between prefill and decode, or throw away cache that an agent is about to need. Evolving Infra aims to fix both problems.

Three design choices

The project describes three ideas.

  • Separate compute from cache. A node can switch between prefill and decode in seconds without restarting. Its cache lives in a pool of host memory that every node can read, so switching roles is cheap.
  • Predict the load. A cost model estimates the prefill and decode work agents will create, including sub-agents and context management. A planner uses it to set the split, and a router uses it to place each request.
  • Manage the cache with agent signals. Context management tells the system what can go. Histories replaced by a summary and the context of finished sub-agents are evicted, while material an agent branch will reuse is kept.

The router sends each request to one prefill node and one decode node, and the planner moves nodes between the two groups as the workload changes.

Run by a coding agent

The repository calls itself agent-native. It includes a rules file and a set of step-by-step skills, so a coding agent such as Claude Code or Codex can deploy the stack, connect an agent harness, reproduce benchmarks and diagnose failed runs. The agent checks each step and reports what it ran.

Everything ships in a runtime image for both x86_64 and aarch64 machines. Reproducing the paper's Claude Code result on a Slurm cluster holds nine GPU nodes for about two hours, so this is not a laptop experiment.

On licensing, the project is MIT-licensed. Its engine and router core derive from SGLang and stay under Apache License 2.0.

How much to trust it yet

The headline figures come from the authors' paper and their chosen workloads, and they are stated as maximums ("up to"). The results will not necessarily hold on your hardware or with your agents. When we saw the project on Hacker News it had barely any attention, so independent testing is still to come.

The cluster scale is also worth noting. Most of the evidence is on very large GPU deployments, and smaller setups may see smaller gains. Still, the central idea is easy to follow: treat agent traffic as its own kind of workload and keep the cache that agents will reuse.

What to watch

  • Will outside teams reproduce the reported gains over NVIDIA Dynamo on their own workloads?
  • Do the benefits hold on smaller clusters than the large GPU setups in the paper?

Try it

If you run inference, skim the repository's architecture notes and benchmarks folder to see whether your agent traffic resembles the tested workloads before investing time.

Sources and links

First reported by github.com. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news