Shan Yang breaks down RL gains: 82% comes from fitting answers into the token budget

Outcome RL raises pass@1 in reasoning models, but pass@k at large k does not grow. Yue et al. showed this in 2025. Shan Yang asks a question that usually stays out of this debate: every pass@1 is measured under a limited token budget, and RL recipes are also trained under such a limit. He compared 20 published RL models, from 1.5B to 32B, with the thinking models they started from, on physics and math tasks at fixed budgets. His conclusion: for the most part, RL teaches a model to reach an answer and to do so more consistently. It rarely unlocks new problems.

Three components of a single gain

The author splits each change in pass@1 into three parts:

• Finishing — the model managed to answer before running out of its token budget.
• Reliability — the answer is correct when the model reaches the end.
• Reach — problems the RL model solves while the base fails all eight attempts, minus problems the base solves while the RL model fails all eight.

This decomposition shows what a gain actually is: a change in behavior or new knowledge.

Where RL helps finish the answer

When the base model often hits the token limit, almost all of the change, both positive and negative, comes from finishing: 82% at 4k and 8k budgets. Give the base model more tokens, and it closes most of that gap on its own. Many thinking models reason at length and simply fail to produce a final answer inside a tight window. RL trains them to fit into the very budget they were trained on.

If the base model has enough tokens, the gain comes from reliability. The RL model gets a correct grade more often, even on problems that both models finish. In other words, RL improves the solution process itself, not just the length of the answer.

Reach: rare new solutions and how they hold up

By the author's definition, RL rarely solves what the base model cannot. On each of the four physics and math sets, reach adds no more than 2.1 percentage points.

The author also checked individual cases. He took 144 physics problems that only an RL model solved, then resampled them or read the solutions. Only 3.5% held up. The remaining "wins" turned out to be chance or flawed reasoning that landed on a guessed answer. In a second check, on a random sample of human-verified new reasoning steps, nine out of ten also appear in samples from the base model. RL more often selects what the base model already knew how to do than invents something new.

A small RL experiment on 8B

In a small RL run on an 8B model, the gain on new problems again came from finishing, while reliability did not change. The author also tried feeding each problem a description of its physical setup. This approach showed no improvement over standard RL.

What this means for practitioners

For those comparing models, the main takeaway is about methodology. A pass@1 table at a single small budget mixes problem-solving skill with the ability to fit within a limit. Results should be checked at several budgets, looking at how much of the gain disappears when the base model gets more tokens. Otherwise, it is easy to mistake the benefit of shorter reasoning for an increase in capability.

For those training models, the picture is sobering. If an RL recipe is squeezed into a tight token limit, you are buying completed answers first and reliability only second. This method has not yet managed to expand the boundaries of what a model can do. It makes sense, then, to compare RL with cheaper options: a larger inference budget or distillation. New knowledge requires other sources of signal, and they should be checked with filters as strict as the author's 144 problems, of which only a handful survived.

You can verify the results yourself. The code, test sets and row-level data are on GitHub (shanyang-me/outcome-rl-buys), and the article's main figures can be recomputed from the published rows without a GPU. Project page: shanyang.me/what-outcome-rl-buys.