AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

TL;DR - AgentOPSD is a critic-free method for turn-level credit assignment in agentic RL, converting sparse outcome rewards into per-turn credit via recursive Bayesian belief updates. It matters because long-horizon multi-turn agent training struggles to identify which few decisions actually determined success.

• Aggregates token-level teacher-student log-probability gaps into turn-level evidence, then recursively updates a Bayesian belief state in log-odds space to reweight turns.
• Pivotal turns are identified through marginal belief revision between consecutive states; the scheme plugs into standard policy optimization with no extra critic and no extra rollouts.
• Evaluated on ALFWorld, WebShop, and Search-QA with Qwen2.5 at 3B and 7B; beats GRPO and strong self-distillation baselines, reaching 89.1% success on ALFWorld with Qwen2.5-7B.
• Ablations credit the gains specifically to turn-level aggregation and history-dependent recursive belief updates, rather than denser supervision alone.