Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling

TL;DR - This study finds that on-policy distillation improves LLM reasoning mainly by making successful outputs more likely with small sampling budgets, rather than expanding the model’s ultimate reasoning capabilities.

• OPD models sustain better avg@K across sampling budgets.
• As K increases, pre-OPD models gradually overtake OPD models on pass@K.
• Training shifts performance toward stronger small-K results at the expense of the large-K capability boundary.
• At pass@1024, OPD makes more previously solvable problems unsolvable than it makes previously unsolvable problems solvable.