$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

TL;DR - β-OPSD generalizes on-policy self-distillation by introducing a tunable KL regularization parameter that balances a reference policy with privileged teacher guidance. It improves training stability and mathematical reasoning performance while retaining efficient distillation-based training.

• Derives the optimal policy as a geometric interpolation between the reference policy and teacher.
• Approximates costly, high-variance policy optimization by mixing reference and teacher token-level logits as distillation targets.
• Uses return-to-go credit assignment to better align token updates with sequence-level objectives.
• Consistently outperforms vanilla OPSD on mathematical reasoning benchmarks.