Together AI launches agent-driven compiler to speed up AI inference

The problem of stale optimization

Most AI models are optimized for specific types of data during deployment. However, real-world traffic is rarely static. For example, Together AI observed that prompt lengths in different countries varied significantly, with one region seeing 97K tokens while another saw 70K.

When traffic patterns shift, the initial optimizations can become inefficient. One customer reported their average input length growing from 18K to 33K tokens in just three days. If a model is tuned for the old data, it may perform poorly on the new requirements.

Additionally, using random data for testing can be misleading. Together AI found that random inputs activated more MoE experts than actual production traffic. This discrepancy means that a configuration that works in a lab might fail in the real world.

How the compiler works

Together Compiler (TKCC) addresses these issues by creating a continual compilation loop. It uses coding agents to rewrite a model's forward pass—the sequence of operations the model takes to process input. The system takes a server launch command, a hardware configuration, and a traffic profile as its starting point.

The agents propose new versions of the forward pass code, which a persistent engine then measures. Unlike traditional systems that restart the entire server for every test, TKCC starts the engine once and swaps the forward function pointer. This allows the system to re-capture CUDA graphs quickly without the 761 seconds of cold startup time seen in other frameworks.

To balance speed and accuracy, the system uses two measurement tiers. A fast tier serves 12 layers for quick screening, while a slow tier serves 60 layers to confirm the gains on real data distributions. This two-tier approach allowed the system to test 8.1x more candidates than a single-tier setup, resulting in a 7.2% faster final solution.

Agent performance and logic

The system relies on multiple coding agents working in parallel and sharing findings on a message board. This high-frequency communication helped make the search for optimal code 4.7x faster in certain tests. The agents are given access to raw engine source code rather than an intermediate representation, allowing them to perform deeper optimizations.

Interestingly, the research showed that naive agents often performed better than human-designed strategies. When agents were simply told to make the forward pass as fast as possible, they beat expert-curated knowledge bases by 6.2x compared to 4.9x in some trials. Human-designed logs also consumed up to 43% of the agents' tool calls, which hindered their efficiency.

Before the agents begin, a cost model ranks different parallelism layouts. This model combines spec-sheet numbers, microbenchmarks, and measured forward passes to account for factors like expert imbalance. On the Kimi K3 model, this pre-ranking helped find a production layout with up to 24% higher throughput.

Benchmarks and results

Together AI tested the compiler on several models, including Kimi K3, GLM 5.3, and Llama 1B. On the MiniMax M3.1 Flash model using NVIDIA B200 hardware, the compiler achieved 1,111 P90 TPS per user. This was a 1.8x improvement over the 633 TPS baseline reported by InferenceX.

The system also showed significant gains on other hardware configurations. For DeepSeek v4.1 Flash on GB300 hardware, it reached 1,205 TPS compared to a 943 TPS baseline. While some configurations saw smaller gains, such as a 4% increase for GLM 5.3 on AMD MI355X, the overall performance remained competitive.

Beyond large language models, the compiler was also tested on scientific models. It sped up 8 different scientific models by 1.5x to 6.4x while maintaining bit-exact outputs. This suggests the technology is versatile enough to handle specialized workloads beyond standard text generation.

Impact on production AI

The primary impact of Together Compiler is its ability to handle the 'drift' in production traffic. Because prompt lengths and request types change constantly, a static optimization will eventually become suboptimal. TKCC ensures that the model stays tuned to the current reality of user behavior.

This automation reduces the need for manual intervention by engineers. By using a persistent engine and hot reloading, the system can update the model's performance characteristics without taking the service offline. This provides a more seamless experience for both the developers maintaining the service and the end users consuming it.

As AI models become more complex, such as Mixture-of-Experts models, the difficulty of manual tuning increases. Together Compiler provides a scalable way to manage these complexities. By leveraging coding agents to navigate the vast space of possible optimizations, it makes high-performance inference more attainable for a broader range of applications.