SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

TL;DR - SEED is a self-evolving RL framework that turns a policy's own completed trajectories into natural-language "hindsight skills" and distills them back as dense token-level supervision, addressing the sparse-reward problem in training long-horizon LLM agents.

• Problem addressed: Outcome-based RL gives only sparse trajectory-level rewards, leaving a supervision gap between episode outcomes and token-level policy learning.
• Core mechanism: The policy doubles as an analyzer that extracts reusable skills (workflows, decisive observations, failure-avoidance rules) from its own on-policy trajectories, so hindsight supervision co-evolves with the policy.
• Distillation signal: Actions are re-scored under ordinary vs. skill-augmented contexts; the resulting probability shift becomes a dense on-policy distillation signal jointly optimized with outcome-based RL.
• Results claimed: Reported gains in performance and sample efficiency across text- and vision-based agentic tasks, with robust generalization to unseen scenarios; code released on GitHub.