DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models

TL;DR - DASH is a training method for reasoning LLMs that reweights token-level self-distillation supervision based on how teacher-student divergence evolves across a rollout, rather than treating every token's divergence identically. It matters because it squeezes better math-reasoning gains out of on-policy self-distillation at zero extra compute.

• Context: RLVR gives sparse sequence-level rewards; on-policy self-distillation (OPSD) densifies this by querying a privileged teacher at student-visited prefixes for token-level distributional supervision.
• Identified gap: standard OPSD applies a uniform coefficient to every local divergence, ignoring token position and the preceding discrepancy history, so it cannot distinguish equal-magnitude divergences arising from different temporal contexts.
• Method: DASH compares each local distillation signal to the sequence-level mean, maps that gap to an adaptive propagation gate, and uses the gates to control backward multi-step aggregation of supervision weights.
• Results: improves over matched vanilla OPSD reruns on all three mathematical reasoning benchmarks at all three model scales tested, reusing distributions OPSD already computes — no extra teacher or student forward passes. Code at github.com/DBtxy/DASH-OPSD.