在线蒸馏也会⼀步错步步错？Relay-OPD让教师关键时刻接棒
Ranking
Overall
79
Content
85
Popularity
65
Observed public metrics from 1 member.
Merged summary
TL;DR - Relay-OPD is an online distillation method that detects when a student model’s reasoning begins to diverge and briefly hands generation to a teacher before returning control. It improves mathematical reasoning accuracy while cutting training trajectory length by more than half.
- A label-free trigger fires when the teacher favors a reflective token such as “Wait,” “But,” or “However,” while none appears among the student’s top-K choices.
- Teacher interventions are deliberately early and limited: the main setup allows two handoffs of three paragraphs each, followed by single-sample distillation on the resulting relay trajectory.
- With a Qwen3-4B teacher and 1.7B student, Relay-OPD averaged 46.96 accuracy across eight math benchmarks, 5.73 points above standard OPD, while reducing average trajectories from 4,658 to 2,296 tokens.
- A speculative-decoding implementation supplies teacher logits and alternates teacher/student generation in one engine, avoiding separate verifiers, reward models, process labels, or correctness labels.
Sources (1)
在线蒸馏也会⼀步错步步错？Relay-OPD让教师关键时刻接棒
Public signals
Hugging Face upvotes 32
TL;DR - Relay-OPD is an online distillation method that detects when a student model’s reasoning begins to diverge and briefly hands generation to a teacher before returning control. It improves mathematical reasoning accuracy while cutting training trajectory length by more than half.
- A label-free trigger fires when the teacher favors a reflective token such as “Wait,” “But,” or “However,” while none appears among the student’s top-K choices.
- Teacher interventions are deliberately early and limited: the main setup allows two handoffs of three paragraphs each, followed by single-sample distillation on the resulting relay trajectory.
- With a Qwen3-4B teacher and 1.7B student, Relay-OPD averaged 46.96 accuracy across eight math benchmarks, 5.73 points above standard OPD, while reducing average trajectories from 4,658 to 2,296 tokens.
- A speculative-decoding implementation supplies teacher logits and alternates teacher/student generation in one engine, avoiding separate verifiers, reward models, process labels, or correctness labels.