SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

TL;DR - SAF-OPD is a framework for stably combining RLVR (response-level verifiable rewards) with on-policy distillation (token-level teacher advantages), fixing the entropy collapse that naive fixed-coefficient fusion causes. It matters because it lets a student model get dense teacher signal without being capped at teacher quality or losing exploration.

• Diagnoses two miscalibrations in fixed-coefficient fusion: a magnitude mismatch (token-level OPD advantages spike past the bounded RLVR advantage and erase it) and a temporal mismatch (sustained full-strength OPD keeps pulling the student toward the teacher).
• SAF applies a four-stage pipeline to the OPD advantage only: sparsify-then-compress for magnitude control, warm-up-then-anneal for temporal control; stages are independently switchable with negligible overhead.
• Instantiated with GRPO as the RLVR method, evaluated on seven math-reasoning and code-generation benchmarks with Qwen3-1.7B/4B/8B.
• Reported gains: avoids entropy collapse and beats fixed-coefficient GRPO+OPD, improving aggregate score by 0.51–2.70% across all six model-domain settings with more stable training.