RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

TL;DR - RTPO is a reinforcement-learning method that trains multi-turn, tool-using LLM agents by updating turns in reverse temporal order. It aims to stabilize agentic RL by improving credit assignment and preventing context mismatch and policy drift across trajectories.

• Represents multi-turn rollouts as sparse reverse trees and aligns each decision with its downstream continuation.
• Unifies three instability sources: rollout-training context mismatch, sparse-reward turn-level credit assignment, and asynchronous policy drift.
• Theoretical results claim elimination of context mismatch and asynchronous drift, reduced credit bias, and convergence to recursive optimality.
• On multi-turn agentic RL benchmarks, RTPO improves over trajectory-level and turn-level baselines by 21.50% and 10.76%, respectively.