AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
Ranking
Overall
74
Content
75
Popularity
71
Observed public metrics from 1 member.
Merged summary
TL;DR - AgentOPSD is a critic-free method for turn-level credit assignment in agentic RL, converting sparse outcome rewards into per-turn credit via recursive Bayesian belief updates. It matters because long-horizon multi-turn agent training struggles to identify which few decisions actually determined success.
- Aggregates token-level teacher-student log-probability gaps into turn-level evidence, then recursively updates a Bayesian belief state in log-odds space to reweight turns.
- Pivotal turns are identified through marginal belief revision between consecutive states; the scheme plugs into standard policy optimization with no extra critic and no extra rollouts.
- Evaluated on ALFWorld, WebShop, and Search-QA with Qwen2.5 at 3B and 7B; beats GRPO and strong self-distillation baselines, reaching 89.1% success on ALFWorld with Qwen2.5-7B.
- Ablations credit the gains specifically to turn-level aggregation and history-dependent recursive belief updates, rather than denser supervision alone.
Sources (1)
AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
Public signals
Hugging Face upvotes 101
TL;DR - AgentOPSD is a critic-free method for turn-level credit assignment in agentic RL, converting sparse outcome rewards into per-turn credit via recursive Bayesian belief updates. It matters because long-horizon multi-turn agent training struggles to identify which few decisions actually determined success.
- Aggregates token-level teacher-student log-probability gaps into turn-level evidence, then recursively updates a Bayesian belief state in log-odds space to reweight turns.
- Pivotal turns are identified through marginal belief revision between consecutive states; the scheme plugs into standard policy optimization with no extra critic and no extra rollouts.
- Evaluated on ALFWorld, WebShop, and Search-QA with Qwen2.5 at 3B and 7B; beats GRPO and strong self-distillation baselines, reaching 89.1% success on ALFWorld with Qwen2.5-7B.
- Ablations credit the gains specifically to turn-level aggregation and history-dependent recursive belief updates, rather than denser supervision alone.