When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents

TL;DR - An empirical study showing that persistent spatial memory in VLM agents silently goes stale as environments change, and that trusting it uncritically is a measurable safety liability. It reframes memory staleness as a safety failure mode rather than a mere accuracy issue.

• Uses a dynamic FrozenLake testbed pairing a staleness-detection task with downstream navigation across three closed-source and three open-weight VLMs, in text and image input modes (1,800 detection runs; 12,000 text-mode navigation episodes over four LLM navigators at a shared 50-seed scale).
• Text solvability doesn't imply visual grounding: models that reliably flag stale entries from text span vision F1 from 0.887 down to 0.067 on identical grids, with the weakest confidently ignoring the image.
• Unaudited stale memory is worse than none: in the primary GPT-4o setting, an agent trusting raw memory dies more than twice as often as the same agent given no memory at all.
• Auditing only partially helps: a transparent read-time filter removes much of the text-mode safety cost, but oracle stale labels add no further significant gain at this grid size, and filtering gives no consistent benefit when visual auditing is unreliable.