SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

TL;DR - SPO++ improves asynchronous reinforcement learning for tool-using agents by aligning advantage normalization with the token-level actor objective and organizing experience by the policy version that generated it. It learns more efficiently than SPO in matched ALFWorld and Math-TIR experiments.

• Standardizes terminal-outcome advantages under the action-token measure, correcting a mismatch between trajectory-level centering and token-mean optimization.
• Groups prompt evidence by its generating policy event rather than by learner receipt order.
• Avoids waiting for sibling rollouts of the same prompt, supporting long, variable-duration agent trajectories.
• Paired ablations identify action-token-measure normalization as the strongest tested component.