On-policy Distillation with Verifiable Reward

TL;DR - OPDVR combines on-policy distillation’s dense token-level guidance with verifiable task-level rewards for LLM post-training, without introducing new hyperparameters. It aims to preserve teacher guidance while preventing incorrect trajectories from receiving positively aligned distillation signals.

• Reformulates sampled-token distillation rewards according to trajectory correctness.
• Uses ReLU gating so correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards.
• Converts sampled-token distillation into an RLVR-compatible objective that can be used with policy-gradient methods such as GRPO.
• Consistently outperforms standard on-policy distillation across six reasoning benchmarks.