Apple identified when on-policy distillation helps model training

Researchers from Apple ML Research have published an analysis of the boundaries of on-policy distillation — a training technique where a teacher model provides per-token guidance to reasoning models. The conclusion: the method can be a powerful tool, but only when applied correctly.

What is on-policy distillation and why is it needed

On-policy distillation is a way to train a new model by giving it very detailed guidance at each step. Imagine: a smart teacher solves a problem aloud, explaining each step, and a student learns not just from the final answer, but from how the teacher reasoned along the way.

This is especially important for reasoning models — models that solve complex problems step by step, unfolding the logic. OpenAI and other labs use similar approaches when training models like o1 and Claude 3.5 Sonnet. At first glance, the more detailed the guidance, the better the student should learn.

When it really helps and when it can hurt

Apple figured out that it's more complex. Key questions remain open:

• Which model to choose as a teacher (stronger is not always better)
• What context to use during self-distillation, when a model learns from itself
• The optimal choice may differ from token to token

The current approach to these questions is usually this: run expensive training, compute for several hours (or days) on GPU, and look at the final metrics. But the problem is that these aggregated indicators hide the truth: at the level of individual tokens, methods often work completely differently.

How Apple proposes to solve this

The research group presented a training-free approach — a way to understand the effectiveness of on-policy distillation WITHOUT expensive experiments. This allows ML engineers to understand whether the method works in their specific scenario before running full model training.

Such tools are critical for large labs: every hour of GPU time costs money, and the ability to predict training results in advance saves resources and development time.

What this means for the ML community

On-policy distillation remains a powerful technique, but this research shows that it cannot be applied mechanically. Different tasks, different models, different data require different solutions — and Apple provided a tool to make these decisions based on evidence, without costly trial-and-error.