SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

TL;DR - SRPO is a post-training framework that has LLMs reflect on completed reasoning trajectories and convert sparse outcome feedback into dense, token-level supervision. It reports strong mathematical reasoning and long-horizon agent performance with substantially lower training compute than scaled supervised fine-tuning.

• Models summarize trajectory errors as concise “reflection patches” that guide subsequent training.
• Reflection-conditioned teacher scores supervise on-policy student rollouts without external critics, separate reward models, or larger teachers.
• A Qwen3-8B model reaches 73.3% on AIME’24 using 8% of the training FLOPs required by scaled supervised fine-tuning.
• Reported success rates include 64.7% on WebShop, 76.8% on ALFWorld, and 31.2% on SWE-Bench-Lite.