‹ back to 2026-08-07

On-Policy Self-Distillation without Any Supervision

arXiv cs.LG LLMs & Foundation Models Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian, Di Fu, Nuno Vasconcelos 2026-08-06

TL;DR - An arXiv paper proposing U-OPSD, an on-policy self-distillation method that post-trains LLMs using only their own generations — no ground truth, environment feedback, or larger teacher model — and still matches or beats supervised baselines like OPSD and GRPO on math reasoning benchmarks.

Pseudo-supervision comes from internal consistency: sample multiple rollouts, then form a pseudo-solution by majority vote gated on a self-consistency threshold.
The teacher distribution is conditioned on the shortest pseudo-solution and distilled into prefixes of the model's longest incorrect completion, targeting cases where the model is confidently wrong.
On AIME24/25, HMMT25, MATH500, and AMC23 with Qwen3 non-thinking mode, it gains 8.5% (4B) and 10.7% (8B) over base models, beating OPSD by 3.2% and 2.3% respectively.
In thinking mode it is roughly on par with OPSD (+0.9% at 4B, tied at 8B) while exceeding GRPO by 0.7% and 1.1% — notable since both baselines use ground-truth signals.

view merged work →