SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

TL;DR - SAPO is an agentic reinforcement-learning method that generates policy and value predictions from one shared autoregressive LLM backbone and requires only a single rollout. It improves task performance while reducing critic memory overhead and iteration time.

• Separately optimizes PPO objectives and auxiliary on-policy SARSA objectives within the shared backbone.
• Uses trajectory-level generalized advantage estimation with lambda-returns and batch normalization for turn-level credit assignment.
• On ALFWorld and WebShop with Qwen2.5-1.5B/7B, SAPO outperformed PPO and GRPO by an average of 15.1 and 12.1 percentage points, respectively.
• Eliminating a separate critic model reduced per-iteration runtime by 33.2% versus PPO.