‹ back to 2026-07-19

Process Reward Informed Tree Rollout for Effective Multi-Turn RL

arXiv cs.CL LLM Agents Xintong Li, Sha Li, Yuwei Zhang, Changlong Yu, Rongmei Lin, Hongye Jin, Shuyi Guan, Xin Liu, Linwei Li, Qingyu Yin, Jingbo Shang 2026-07-17

TL;DR - PATR (Process-Scorer Guided Adaptive Tree Rollout) is a rollout framework for multi-turn RL that structures agentic trajectories as trees and uses process feedback to branch from promising states, improving exploration efficiency under a fixed training budget.

Reframes exploration in multi-turn agent RL as a branching decision: treats each turn as a decision point in a trajectory tree rather than sampling independent complete trajectories (as in GRPO/RLOO).
Uses task-appropriate process feedback to score partial trajectories, selectively branching from promising states, reusing shared prefixes, and conservatively pruning degenerate paths to cut wasted sampling.
Rollout groups stay compatible with standard policy optimization, so it integrates without altering the optimization method.
Reports gains of up to +5.0 points on SWE-Bench and +9.3 points on FrozenLake, with SWE-Bench noted as largely unexplored by prior tree-rollout agent RL work.

view merged work →