2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation

TL;DR - 2AM separates memory from robotic control: a multimodal agent retains interaction history and steers an episodically stateless, RGB-only vision-language-action model through language and optional 2D spatial hints. On LIBERO-Mem, it achieves 76.3% average completion, suggesting that richer agent-policy interfaces can enable long-horizon manipulation without embedding memory in the action model or relying on depth and geometric planning.

• The multimodal agent is the sole holder of task memory, while one action model executes all task-relevant motion.
• The agent translates history into subtask instructions plus optional 2D grasp, placement, and movement hints at multiple time scales.
• Training uses structured hint labels, condition dropout, spatial noise, and temporal jitter to make execution robust to imperfect guidance.
• 2AM reports 76.3% average completion, 63.0% relaxed success, and 11.8% strict success, versus 14.8% completion for the strongest reported baseline.