Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
Ranking
Overall
91
Content
100
Popularity
69
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper finds that on-policy distillation gains largely come from suppressing low-probability tokens rather than learning from teacher-provided advantages. It introduces On-Policy Self-Adaptation (OPSA), a teacher-free alternative that substantially improves mathematical reasoning performance.
- Teacher supervision during on-policy distillation is substantially noisy, with noise increasing as teacher scale grows, yet removing it produces comparable student performance.
- A fixed negative advantage matches teacher-provided advantages, suggesting the core benefit is suppression of low-probability tail tokens.
- OPSA applies entropy-adaptive negative advantages at high-entropy positions, suppressing tail tokens and redistributing probability among likely head tokens.
- On AIME24, OPSA improves Qwen3-1.7B Avg@32 by 35.41 points and outperforms on-policy distillation by 16.77 points.
Sources (1)
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
Public signals
Hugging Face upvotes 96
TL;DR - This paper finds that on-policy distillation gains largely come from suppressing low-probability tokens rather than learning from teacher-provided advantages. It introduces On-Policy Self-Adaptation (OPSA), a teacher-free alternative that substantially improves mathematical reasoning performance.
- Teacher supervision during on-policy distillation is substantially noisy, with noise increasing as teacher scale grows, yet removing it produces comparable student performance.
- A fixed negative advantage matches teacher-provided advantages, suggesting the core benefit is suppression of low-probability tail tokens.
- OPSA applies entropy-adaptive negative advantages at high-entropy positions, suppressing tail tokens and redistributing probability among likely head tokens.
- On AIME24, OPSA improves Qwen3-1.7B Avg@32 by 35.41 points and outperforms on-policy distillation by 16.77 points.