TTPO: Test-Time Policy Optimization

TL;DR - TTPO is a label-free test-time policy optimization method that combines self-distillation on majority-agreeing rollouts with grouped reinforcement learning on disagreeing ones. It matches label-supervised OPSD across five competition-level mathematical reasoning benchmarks while improving Qwen3-1.7B test-time accuracy from 38.0% to 45.2%.

• Uses an asymmetric objective designed to remain useful even when majority-vote pseudo-labels are incorrect.
• Distills agreeing rollouts via on-policy self-distillation and penalizes disagreeing rollouts through grouped reinforcement learning.
• Applies token-level selection to down-weight converged positions and penalize only confident errors.
• Reports strong label-free gains, including 25.2%–36.4% improvements without thinking and cross-task generalization.