$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
Ranking
Overall
80
Content
85
Popularity
68
Observed public metrics from 1 member.
Merged summary
TL;DR - β-OPSD generalizes on-policy self-distillation by introducing a tunable KL regularization parameter that balances a reference policy with privileged teacher guidance. It improves training stability and mathematical reasoning performance while retaining efficient distillation-based training.
- Derives the optimal policy as a geometric interpolation between the reference policy and teacher.
- Approximates costly, high-variance policy optimization by mixing reference and teacher token-level logits as distillation targets.
- Uses return-to-go credit assignment to better align token updates with sequence-level objectives.
- Consistently outperforms vanilla OPSD on mathematical reasoning benchmarks.
Sources (1)
$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
Public signals
Hugging Face upvotes 24
TL;DR - β-OPSD generalizes on-policy self-distillation by introducing a tunable KL regularization parameter that balances a reference policy with privileged teacher guidance. It improves training stability and mathematical reasoning performance while retaining efficient distillation-based training.
- Derives the optimal policy as a geometric interpolation between the reference policy and teacher.
- Approximates costly, high-variance policy optimization by mixing reference and teacher token-level logits as distillation targets.
- Uses return-to-go credit assignment to better align token updates with sequence-level objectives.
- Consistently outperforms vanilla OPSD on mathematical reasoning benchmarks.