SkyRL boosts Mistral Large 4 math scores using 8-node reinforcement learning

A day-zero performance jump

The SkyRL team at NovaSky recently published a report showing significant progress on the Mistral Large 4 model. They managed to increase the model's AIME 2024 average@8 score from roughly 64% to about 75%. This improvement occurred in only 75 training steps, making it one of the first results published shortly after the model's weights became available.

AIME 2024 is a set of competition-level mathematics problems. The 'average@8' metric means the model was given eight attempts at each problem, with the final score being the average of those completions. Because the move was 11 points on a sample of 240 completions, the result is statistically significant and not just a result of random noise.

As OrcaRouter reported, this is a notable achievement for a

Technical choices for the model

The success of this run relied heavily on the specific choice of the NVFP4 quantized checkpoint. This version of the model is 610.1 GB across 32 shards, which is significantly smaller than the 1,060.6 GB FP8 parent model. By using this smaller footprint, the team was able to fit a trillion-parameter mixture of experts model onto eight nodes while still leaving enough memory for activations and the KV cache.

This configuration is what makes the experiment viable on a smaller scale of hardware. It allows the base model to remain in its quantized form while the LoRA adapters receive the gradients. This setup keeps the optimizer state small, which is a critical requirement for training such a large model on rented infrastructure without needing a massive data center.

Efficiency and benchmark nuances

Beyond the accuracy scores on math problems, the reinforcement learning process also impacted the model's output length. The report notes that the model produced about 30% fewer tokens per task after the RL training. For many buyers, this reduction in verbosity is a more practical improvement than the specific percentage points gained on a math benchmark, as it suggests a more direct and concise reasoning path.

It is also important to distinguish these results from other types of improvements. For example, a different run showed a 13-point move on Terminal-Bench, while this run showed an 11-point move on AIME. These are not the same type of achievement expressed in different units; they represent different tasks with different reward structures that happen to result in similar numerical gains.

Data baselines and reproduction

The 64% baseline used for the AIME score is a measurement taken by the SkyRL team on their own harness. Because there is no official Mistral-published score for this model on this specific benchmark, the delta is internally consistent but externally unanchored. This means you cannot compare the starting point against a vendor figure, as no such official figure exists for the public to reference.

To replicate these results, the NVFP4 checkpoint is currently available as an ungated download without a Hugging Face account gate. A practical shopping list for reproduction includes the 610 GB checkpoint, roughly 32 Blackwell-class GPUs across eight nodes, and a vLLM build with a working NVFP4 path. The Mistral Ecosystem Open-Weight License permits these types of private fine-tuning and evaluation runs for commercial purposes.

Infrastructure and training loops

The technical setup reveals specific details about how the training loop was constructed. While the note provides the hardware and checkpoint names, it functions primarily as a recipe announcement rather than a complete reproduction package. The team used vLLM to generate the rollouts, while the training half was handled by either FSDP2 or Megatron. This distinction is important because it shows the specific software components required to manage the heavy lifting of a trillion-parameter mixture of experts model during the post-training phase.

Because the report lacks certain specifics like the adapter rank or the exact learning curve, it is best viewed as a proof of concept. It confirms that these large models are responsive to reinforcement learning when using the right hardware and quantization. While it does not provide every variable needed for a perfect replication, it successfully demonstrates the feasibility of using rented Blackwell parts to achieve measurable gains on complex reasoning tasks in a very short timeframe.

Scaling the reward signal

Reinforcement learning on verifiable mathematics is considered the easiest scenario for researchers because the reward is objective. In this specific case, the reward requires no human oversight, no complex rubric, and no external judge model to evaluate the output. A simple script can check if the final answer is correct, which keeps the marginal cost of each rollout limited to the cost of inference. This makes it an ideal first experiment for testing how a new checkpoint responds to a clear and verifiable training signal.

Moving beyond math to more open-ended tasks introduces significant complexity because the reward signal becomes much harder to manage. If a verifier must be a judge model rather than a script, that model must be able to survive being called at massive rollout volumes. This requires a stable interface capable of handling one endpoint over hundreds of models, passing through provider list prices without markups, and managing failovers when a single judge endpoint hits rate limits. Watching for these three things—the publication of the reward definition, a second non-math task, and an independent evaluation—will determine if these results can be scaled.