SkyRL and vLLM Fine-Tune Mistral Large 4 with RL: AIME Score Rises from 64% to 75%

The vLLM project team and the NovaSky AI lab report that their SkyRL framework supports reinforcement learning (RL) for Mistral Large 4 from day one. A cluster of 8 Nvidia GB200 nodes raised the model's AIME 2024 score from roughly 64% to 75% in 75 training steps. AIME 2024 is a set of 30 problems from the American math olympiad, where the answer is always an integer. The avg@8 metric means the model makes 8 attempts at each problem, and the final share of correct answers is averaged. This makes the result less dependent on luck in a single attempt.

What Day-0 means and why it matters

Day-0 means support is ready on the day the model is released. Usually a new large model can at first only be run, and fine-tuning has to wait until the community writes code for its architecture. Here, the team says the SkyRL and vLLM stack worked with Mistral Large 4 right away.

Reinforcement learning works like this. The model receives a problem and writes several candidate solutions. These candidates are called rollouts. Each rollout is then checked: does the answer match the reference? The model gets a reward for a correct answer, and the weights shift so that successful chains of reasoning become more likely. For math, this check is cheap and unambiguous, which is why AIME and similar problem sets are often used for RL.

Who does what in the stack

The main engineering idea is the division of labor. The two stages of the loop load the hardware differently, and a separate tool runs each one:

• vLLM generates the rollouts. It is an engine for fast inference, meaning running the model to produce answers. Generation takes most of the time in an RL loop, so it needs a fast server.
• FSDP2 or Megatron handle training. FSDP2 is the second version of the PyTorch approach in which parameters and gradients are sharded across GPUs. Megatron is Nvidia's framework for training very large models. SkyRL lets you choose either one.
• SkyRL from NovaSky AI ties everything into one pipeline: it takes rollouts from vLLM, computes rewards, and passes the data to training.

In practice, this means an engineer does not have to write the glue between the generation engine and the training code. They take the ready-made stack and plug in a model.

Why LoRA on a 4-bit checkpoint

The post names two techniques that make the process cheaper.

The first is LoRA. Instead of updating all the weights of a huge model, the team trains small additional matrices that attach on top of it. They take up a small fraction of the memory the model itself needs, and Mistral Large 4 stays frozen.

The second is a checkpoint in the NVFP4 format. This is a 4-bit number representation that Nvidia supports on the Blackwell architecture, which includes GB200. Weights in this format take up several times less space than in 16-bit. A large model fits on fewer nodes, and generation runs faster.

The result is a modest setup by large-model standards: 8 nodes, 75 steps, and a gain of about 11 percentage points on a single benchmark.

How to read this result

The figures come from the vLLM project's own announcement, and there is no independent verification yet. The gain on AIME 2024 shows that the RL loop works on this model and hardware. It does not prove that the model got better overall. On math problems with unambiguous verification, RL usually has a noticeable effect, but on open-ended tasks (writing, agentic scenarios, code in a large repository) the result has to be measured separately.

For engineers, the takeaway is this. Fine-tuning a large open model with reinforcement learning has become easier: a ready-made stack of open components works from the first day of the release, and LoRA and NVFP4 lower the hardware requirements. For companies with their own tasks that have automatic answer checking, this is a way to tune a model to their needs without full retraining. What remains is to test on your own data how many of those 11 points you can reproduce.