SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL
Ranking
Overall
88
Content
100
Popularity
59
Observed public metrics from 1 member.
Merged summary
TL;DR - SPO++ improves asynchronous reinforcement learning for tool-using agents by aligning advantage normalization with the token-level actor objective and organizing experience by the policy version that generated it. It learns more efficiently than SPO in matched ALFWorld and Math-TIR experiments.
- Standardizes terminal-outcome advantages under the action-token measure, correcting a mismatch between trajectory-level centering and token-mean optimization.
- Groups prompt evidence by its generating policy event rather than by learner receipt order.
- Avoids waiting for sibling rollouts of the same prompt, supporting long, variable-duration agent trajectories.
- Paired ablations identify action-token-measure normalization as the strongest tested component.
Sources (1)
SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL
Public signals
Semantic Scholar citations 1 · Semantic Scholar influential citations 0
TL;DR - SPO++ improves asynchronous reinforcement learning for tool-using agents by aligning advantage normalization with the token-level actor objective and organizing experience by the policy version that generated it. It learns more efficiently than SPO in matched ALFWorld and Math-TIR experiments.
- Standardizes terminal-outcome advantages under the action-token measure, correcting a mismatch between trajectory-level centering and token-mean optimization.
- Groups prompt evidence by its generating policy event rather than by learner receipt order.
- Avoids waiting for sibling rollouts of the same prompt, supporting long, variable-duration agent trajectories.
- Paired ablations identify action-token-measure normalization as the strongest tested component.