The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

TL;DR - This paper studies how foundation-model agents acquire, refine, and combine long-horizon planning abilities in a controlled multi-turn environment. It identifies training-data and distillation strategies that improve generalization while exposing sources of error and capability interference.

• Explicit chain-of-thought state-transition modeling improves long-horizon generalization; atomic skills alone do not enable composition, but limited long-horizon data can.
• Suboptimal training trajectories substantially degrade performance as errors compound over longer horizons.
• On-policy distillation (OPD) is effective across more low-quality and long-horizon settings than GRPO, owing to more consistent update directions.
• Multi-teacher OPD can integrate compatible or partially shared planning patterns, while conflicting patterns cause severe interference.