Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

TL;DR - This paper finds that on-policy distillation gains largely come from suppressing low-probability tokens rather than learning from teacher-provided advantages. It introduces On-Policy Self-Adaptation (OPSA), a teacher-free alternative that substantially improves mathematical reasoning performance.

• Teacher supervision during on-policy distillation is substantially noisy, with noise increasing as teacher scale grows, yet removing it produces comparable student performance.
• A fixed negative advantage matches teacher-provided advantages, suggesting the core benefit is suppression of low-probability tail tokens.
• OPSA applies entropy-adaptive negative advantages at high-entropy positions, suppressing tail tokens and redistributing probability among likely head tokens.
• On AIME24, OPSA improves Qwen3-1.7B Avg@32 by 35.41 points and outperforms on-policy distillation by 16.77 points.