MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents

TL;DR - MemOPD is an on-policy distillation method for long-horizon LLM agents that use compact memory, fixing a subtle mismatch where the teacher scores student actions under a rewritten context the student never actually saw. It matters because valid dense teacher supervision substantially outperforms sparse-reward PPO for learning what an agent should retain in memory.

• Core problem: memory compression rewrites context between invocations, so flattening rollouts into a persistent history makes actions "on-policy by provenance, but not by state," invalidating teacher scoring.
• Method: record each invocation's inputs and sampled outputs, restore original token positions and causal visibility, and pack reconstructed invocations for efficient teacher scoring; teacher gives full-vocabulary supervision at sampled action positions while PPO retains the final task objective.
• Results: +7.0% F1 over persistent-history teacher scoring in a matched control; MemOPD-3B improves F1 over PPO by up to 416.2%; packing gives up to 1.63x actor-computation speedup during training.
• Code released at github.com/TPssp/MemOPD.