SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Ranking
Overall
80
Content
95
Popularity
45
Observed public metrics from 1 member.
Merged summary
TL;DR - SAPO is an agentic reinforcement-learning method that generates policy and value predictions from one shared autoregressive LLM backbone and requires only a single rollout. It improves task performance while reducing critic memory overhead and iteration time.
- Separately optimizes PPO objectives and auxiliary on-policy SARSA objectives within the shared backbone.
- Uses trajectory-level generalized advantage estimation with lambda-returns and batch normalization for turn-level credit assignment.
- On ALFWorld and WebShop with Qwen2.5-1.5B/7B, SAPO outperformed PPO and GRPO by an average of 15.1 and 12.1 percentage points, respectively.
- Eliminating a separate critic model reduced per-iteration runtime by 33.2% versus PPO.
Sources (1)
SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Public signals
Hugging Face upvotes 0
TL;DR - SAPO is an agentic reinforcement-learning method that generates policy and value predictions from one shared autoregressive LLM backbone and requires only a single rollout. It improves task performance while reducing critic memory overhead and iteration time.
- Separately optimizes PPO objectives and auxiliary on-policy SARSA objectives within the shared backbone.
- Uses trajectory-level generalized advantage estimation with lambda-returns and batch normalization for turn-level credit assignment.
- On ALFWorld and WebShop with Qwen2.5-1.5B/7B, SAPO outperformed PPO and GRPO by an average of 15.1 and 12.1 percentage points, respectively.
- Eliminating a separate critic model reduced per-iteration runtime by 33.2% versus PPO.