Self-Improving Large Language Models via Progressive Experience Evolution

TL;DR - SPEE (Self-Progressive Experience Evolution) is a post-training framework that bridges test-time experience extraction and training-time policy optimization by adding an explicit "experience distillation" stage, so transient interaction experience becomes persistent model parameters. It matters because it addresses a structural gap in self-improving LLM pipelines rather than just tuning an existing one.

• Two-stage design: explicit experience evolution (reflect on multi-interaction trajectories to extract, verify, and evolve transferable experience) followed by implicit reward-driven RL policy optimization that explores new strategies using the internalized priors.
• Experience is internalized via privilege-guided On-Policy Self-Distillation (OPSD), the intermediate mechanism the authors argue is missing from existing paradigms.
• A continuously evolving global experience pool consolidates both successful and failed trajectories, filters low-utility experience, and mitigates post-hoc rationalization from individual trajectories.
• Evaluated on five mathematical reasoning benchmarks across three model scales, reported to consistently beat both test-time and training-time self-evolution baselines; code released at github.com/rrrsj/SPEE.