SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
Ranking
Overall
84
Content
95
Popularity
59
Observed public metrics from 1 member.
Merged summary
TL;DR - SRPO is a post-training framework that has LLMs reflect on completed reasoning trajectories and convert sparse outcome feedback into dense, token-level supervision. It reports strong mathematical reasoning and long-horizon agent performance with substantially lower training compute than scaled supervised fine-tuning.
- Models summarize trajectory errors as concise “reflection patches” that guide subsequent training.
- Reflection-conditioned teacher scores supervise on-policy student rollouts without external critics, separate reward models, or larger teachers.
- A Qwen3-8B model reaches 73.3% on AIME’24 using 8% of the training FLOPs required by scaled supervised fine-tuning.
- Reported success rates include 64.7% on WebShop, 76.8% on ALFWorld, and 31.2% on SWE-Bench-Lite.
Sources (1)
SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
Public signals
Hugging Face upvotes 1
TL;DR - SRPO is a post-training framework that has LLMs reflect on completed reasoning trajectories and convert sparse outcome feedback into dense, token-level supervision. It reports strong mathematical reasoning and long-horizon agent performance with substantially lower training compute than scaled supervised fine-tuning.
- Models summarize trajectory errors as concise “reflection patches” that guide subsequent training.
- Reflection-conditioned teacher scores supervise on-policy student rollouts without external critics, separate reward models, or larger teachers.
- A Qwen3-8B model reaches 73.3% on AIME’24 using 8% of the training FLOPs required by scaled supervised fine-tuning.
- Reported success rates include 64.7% on WebShop, 76.8% on ALFWorld, and 31.2% on SWE-Bench-Lite.