Analysis reveals diverse routing strategies across major MoE models

The diversity of MoE architecture

Mixture-of-Experts (MoE) models are designed to run efficiently by only activating a small portion of their total parameters for any given input. While many models share the core concept of token-choice routing, a recent analysis by marktechpost.com shows they differ significantly in their specific configurations. These models vary across five primary knobs: expert count, experts per token, shared experts, scoring functions, and load-balancing methods.

The scale of these models varies wildly across the landscape. For instance, the Qwen3.6-35B-A3B model features 35B total parameters with only 3B active, while the DeepSeek-V4-Pro reaches a massive 1.6T total parameters with 49B active. These differences mean that 'active parameters' is not a uniform metric, as the underlying routing logic changes how those parameters are accessed and utilized by the system.

Researchers analyzed 10 different models to map out these variations. They found that expert counts have grown from as few as 8 in the Mixtral 8x7B model to as many as 384 in the Kimi K2 and DeepSeek-V4-Pro. This expansion allows for more specialized knowledge storage within the model while attempting to maintain manageable computational costs during inference.

How different models route tokens

Routing determines which specific experts receive a token's data. DeepSeek-V3 uses a sigmoid function to score 256 routed experts and then normalizes those scores. It also employs a per-expert bias to steer the selection of the top-8 experts without altering the mixing weights. This specific approach allows the model to keep four groups of experts prioritized while ensuring the token reaches at most four nodes.

DeepSeek-V4-Pro introduces further complexity by using a sqrtsoftplus score function and hash routing in its first three layers. Hash routing creates a fixed map between tokens and experts, which can simplify certain operations. It also maintains a bias-based balancing method known as noaux_tc. These choices reflect a move toward more sophisticated control over how data flows through the model's various expert layers.

Other models take different paths toward efficiency. Llama 4 Maverick, for example, alternates between dense and MoE layers. In its MoE layers, it uses a top-1 design where a token goes to one shared expert and only one of 128 routed experts. This design echoes the Switch Transformer architecture, focusing on extreme sparsity where only a tiny fraction of the 400B total parameters are active at any one time.

Shared experts and balancing methods

A shared expert is a component that runs for every single token, providing a dense path of information. The analysis shows that seven of the ten models studied keep at least one shared expert. Models like Qwen3.6-35B-A3B and DeepSeek-V3 use this to ensure that core knowledge is always processed, while models like Mixtral and gpt-oss omit shared experts entirely to prioritize different architectural goals.

Load balancing is another critical factor, as it prevents certain experts from being overworked while others remain idle. Qwen3 uses a global-batch balancing loss with a coefficient of 0.001 to distribute the workload. In contrast, the DeepSeek family often uses the noaux_tc method, which is a loss-free balance routing. These methods ensure that the model utilizes its available hardware resources effectively during training and inference.

The transparency of these methods varies between developers. While some models provide full details in their configuration files, others are less open. For example, the configurations for gpt-oss and Llama 4 do not disclose their specific balancing methods. This lack of information makes it harder for external researchers to perfectly replicate the performance characteristics of these specific models without internal documentation.

Hardware and performance implications

The way a model routes tokens directly impacts the hardware required to run it. Models with more experts per layer generally require more weights to be resident in memory and generate more all-to-all traffic between processors. This can increase the complexity of the networking required to keep the model running smoothly across multiple GPUs.

Conversely, lowering the top-k value can trim the amount of compute required per token. DeepSeek-V4-Pro, for instance, lowered its top-k from 8 to 6 to manage this balance. The analysis found that Kimi K2 demonstrated that higher sparsity could actually lower the loss at a fixed active compute level, suggesting that more experts might be beneficial if managed correctly. This highlights the ongoing trade-off between model capacity and computational speed.

Memory optimization also plays a role in accessibility. The gpt-oss-120b model uses MXFP4 weight shipping, which allows a 117B parameter model to fit onto a single 80 GB GPU. By combining specific routing choices with specialized data formats, developers can make very large models runnable on more standard hardware. Users must check these specific configuration fields before sizing their GPU clusters for deployment.

Future trends in MoE scaling

The data suggests a trend toward larger expert counts and more sophisticated routing functions. We are seeing a shift from the 8-expert models of the past toward systems with hundreds of experts, such as the 384 experts found in the DeepSeek-V4-Pro. This allows for a more granular distribution of knowledge, where different experts can become highly specialized in niche topics or tasks.

The adoption of the DeepSeekV3ForCausalLM architecture class is also becoming more common. Kimi K2 utilizes this architecture along with the noaux_tc router, indicating that specific successful designs are being adopted by other developers. This standardization helps the community understand how to scale models effectively while maintaining high performance across different tasks.

Moving forward, the industry will likely continue to experiment with the balance between shared and routed experts. As models get larger, the challenge will be minimizing communication overhead while maximizing the utility of each parameter. The variety of routing strategies currently in use shows that there is no single 'correct' way to build an MoE model, but rather a variety of trade-offs based on the desired balance of speed, memory, and accuracy.