在线蒸馏也会⼀步错步步错？Relay-OPD让教师关键时刻接棒

TL;DR - Relay-OPD is an online distillation method that detects when a student model’s reasoning begins to diverge and briefly hands generation to a teacher before returning control. It improves mathematical reasoning accuracy while cutting training trajectory length by more than half.

• A label-free trigger fires when the teacher favors a reflective token such as “Wait,” “But,” or “However,” while none appears among the student’s top-K choices.
• Teacher interventions are deliberately early and limited: the main setup allows two handoffs of three paragraphs each, followed by single-sample distillation on the resulting relay trajectory.
• With a Qwen3-4B teacher and 1.7B student, Relay-OPD averaged 46.96 accuracy across eight math benchmarks, 5.73 points above standard OPD, while reducing average trajectories from 4,658 to 2,296 tokens.
• A speculative-decoding implementation supplies teacher logits and alternates teacher/student generation in one engine, avoiding separate verifiers, reward models, process labels, or correctness labels.