On-Policy Delta Distillation
Merged summary
TL;DR - A new post-training method, On-Policy Delta Distillation (OPD²), improves how reasoning capabilities are transferred from teacher to student LLMs by supervising on a "delta signal" rather than imitating the teacher's raw output distribution. It matters because it makes on-policy distillation more effective, letting reasoning models reach strong performance with only brief post-training.
- Introduces the delta signal as the distillation reward: the difference between a reasoning-tuned teacher and its pre-instruction-tuning base model, isolating changes induced by reasoning tuning.
- Frames this within on-policy distillation, which sidesteps reward-model constraints in RL post-training by offering token-level teacher supervision.
- Reports that OPD² consistently outperforms conventional on-policy distillation across mathematics, science, and code-reasoning benchmarks.
- Claims strong results with only a short post-training period; code is promised at github.com/naver-ai/opd2. (Specific quantitative results aren't provided in the abstract.)
Sources (1)
On-Policy Delta Distillation
TL;DR - A new post-training method, On-Policy Delta Distillation (OPD²), improves how reasoning capabilities are transferred from teacher to student LLMs by supervising on a "delta signal" rather than imitating the teacher's raw output distribution. It matters because it makes on-policy distillation more effective, letting reasoning models reach strong performance with only brief post-training.
- Introduces the delta signal as the distillation reward: the difference between a reasoning-tuned teacher and its pre-instruction-tuning base model, isolating changes induced by reasoning tuning.
- Frames this within on-policy distillation, which sidesteps reward-model constraints in RL post-training by offering token-level teacher supervision.
- Reports that OPD² consistently outperforms conventional on-policy distillation across mathematics, science, and code-reasoning benchmarks.
- Claims strong results with only a short post-training period; code is promised at github.com/naver-ai/opd2. (Specific quantitative results aren't provided in the abstract.)