On-policy Distillation with Verifiable Reward
Ranking
Overall
83
Content
90
Popularity
67
Observed public metrics from 1 member.
Merged summary
TL;DR - OPDVR combines on-policy distillation’s dense token-level guidance with verifiable task-level rewards for LLM post-training, without introducing new hyperparameters. It aims to preserve teacher guidance while preventing incorrect trajectories from receiving positively aligned distillation signals.
- Reformulates sampled-token distillation rewards according to trajectory correctness.
- Uses ReLU gating so correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards.
- Converts sampled-token distillation into an RLVR-compatible objective that can be used with policy-gradient methods such as GRPO.
- Consistently outperforms standard on-policy distillation across six reasoning benchmarks.
Sources (1)
On-policy Distillation with Verifiable Reward
Public signals
Hugging Face upvotes 17
TL;DR - OPDVR combines on-policy distillation’s dense token-level guidance with verifiable task-level rewards for LLM post-training, without introducing new hyperparameters. It aims to preserve teacher guidance while preventing incorrect trajectories from receiving positively aligned distillation signals.
- Reformulates sampled-token distillation rewards according to trajectory correctness.
- Uses ReLU gating so correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards.
- Converts sampled-token distillation into an RLVR-compatible objective that can be used with policy-gradient methods such as GRPO.
- Consistently outperforms standard on-policy distillation across six reasoning benchmarks.