When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents
Ranking
Overall
73
Content
80
Popularity
57
Observed public metrics from 1 member.
Merged summary
TL;DR - An empirical study showing that persistent spatial memory in VLM agents silently goes stale as environments change, and that trusting it uncritically is a measurable safety liability. It reframes memory staleness as a safety failure mode rather than a mere accuracy issue.
- Uses a dynamic FrozenLake testbed pairing a staleness-detection task with downstream navigation across three closed-source and three open-weight VLMs, in text and image input modes (1,800 detection runs; 12,000 text-mode navigation episodes over four LLM navigators at a shared 50-seed scale).
- Text solvability doesn't imply visual grounding: models that reliably flag stale entries from text span vision F1 from 0.887 down to 0.067 on identical grids, with the weakest confidently ignoring the image.
- Unaudited stale memory is worse than none: in the primary GPT-4o setting, an agent trusting raw memory dies more than twice as often as the same agent given no memory at all.
- Auditing only partially helps: a transparent read-time filter removes much of the text-mode safety cost, but oracle stale labels add no further significant gain at this grid size, and filtering gives no consistent benefit when visual auditing is unreliable.
Sources (1)
When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents
Public signals
Hugging Face upvotes 17
TL;DR - An empirical study showing that persistent spatial memory in VLM agents silently goes stale as environments change, and that trusting it uncritically is a measurable safety liability. It reframes memory staleness as a safety failure mode rather than a mere accuracy issue.
- Uses a dynamic FrozenLake testbed pairing a staleness-detection task with downstream navigation across three closed-source and three open-weight VLMs, in text and image input modes (1,800 detection runs; 12,000 text-mode navigation episodes over four LLM navigators at a shared 50-seed scale).
- Text solvability doesn't imply visual grounding: models that reliably flag stale entries from text span vision F1 from 0.887 down to 0.067 on identical grids, with the weakest confidently ignoring the image.
- Unaudited stale memory is worse than none: in the primary GPT-4o setting, an agent trusting raw memory dies more than twice as often as the same agent given no memory at all.
- Auditing only partially helps: a transparent read-time filter removes much of the text-mode safety cost, but oracle stale labels add no further significant gain at this grid size, and filtering gives no consistent benefit when visual auditing is unreliable.