Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory

TL;DR - A training-free framework, Agent Memory Distillation (AMD), transfers structured memory from a large teacher agent to small (4B–8B) student LLM agents, substantially improving tool-use accuracy without any weight updates. It matters because small agents typically can't bootstrap their own successful trajectories for memory-based self-improvement.

• Builds three complementary memory types from successful teacher trajectories: Workflow (task-level strategies), Subtask (intermediate-granularity behavioral examples), and Function (per-function calling conventions and pitfalls).
• Injection is hybrid: Workflow and Subtask memories are added proactively at task start, while Function memory is retrieved reactively only when a tool call errors.
• With GPT-5-mini as teacher, average accuracy gains were 27.2 percentage points on AppWorld, 11.2p on BFCL V3, and 3.4p on ToolSandbox across four student models, beating existing memory-based baselines.
• Ablations attribute the largest share of gains to Subtask memory; effectiveness depends on both teacher capability and teacher–student compatibility, with 4B students benefiting most.