Know It, Act on It: Investigating Memory Utilization in LLM Personalization
Ranking
Overall
68
Content
80
Popularity
40
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv study separating whether personalized LLM agents remember user preferences from whether they act on them, finding a large "know-but-don't-act" gap. It matters because memory benchmarks that only test recall overstate real personalization ability.
- Introduces a decoupled evaluation paradigm: paired "Know" (recall) and "Act" (behavioral) tests administered on the same user preference.
- Large-scale setup: 16 systems, five memory architectures, 1,000 preferences embedded at three levels of expression strength.
- Agents frequently pass recall yet fail to reflect the same preference in the paired behavioral scenario — a knowledge utilization failure, not a retrieval failure.
- Memory architectures narrow but don't close the gap; utilization is weakest for health and therapy preferences, where failure carries the highest real-world stakes.
Sources (1)
Know It, Act on It: Investigating Memory Utilization in LLM Personalization
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - An arXiv study separating whether personalized LLM agents remember user preferences from whether they act on them, finding a large "know-but-don't-act" gap. It matters because memory benchmarks that only test recall overstate real personalization ability.
- Introduces a decoupled evaluation paradigm: paired "Know" (recall) and "Act" (behavioral) tests administered on the same user preference.
- Large-scale setup: 16 systems, five memory architectures, 1,000 preferences embedded at three levels of expression strength.
- Agents frequently pass recall yet fail to reflect the same preference in the paired behavioral scenario — a knowledge utilization failure, not a retrieval failure.
- Memory architectures narrow but don't close the gap; utilization is weakest for health and therapy preferences, where failure carries the highest real-world stakes.