RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - RoMeRL is a reinforcement-learning memory framework for self-evolving LLM agents that replaces an ever-growing trajectory-indexed utility space with a fixed-dimensional per-task memory state, concentrating sparse feedback and limiting reward contamination. It matters because it makes learned agent memory both denser in feedback and cheaper to maintain over long-horizon interaction.
- Diagnoses two coupled failure modes: trajectory-indexed utilities disperse limited feedback across an expanding state space, and trajectory-level rewards jointly credited to co-retrieved memories push irrelevant experiences into a "memory-reward trap."
- Proposes a reduced-order parameterization: a fixed set of semantic coordinates factorized by outcome polarity and memory dynamics, whose contents are updated or replaced rather than appended, bounding the utility support.
- Theoretical results argue the reduced-order form raises average feedback per utility coordinate and characterize steady-state occupancy of erroneous coordinates under a generic coordinate-transition model.
- On ALFWorld and LifelongAgentBench: improved task performance, Cold-Q ratio down 80.0%, feedback density up ~6.0x, maintained memory size down 84.4%, and LLM calls cut 21.1%; code released on GitHub.
Sources (1)
RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States
TL;DR - RoMeRL is a reinforcement-learning memory framework for self-evolving LLM agents that replaces an ever-growing trajectory-indexed utility space with a fixed-dimensional per-task memory state, concentrating sparse feedback and limiting reward contamination. It matters because it makes learned agent memory both denser in feedback and cheaper to maintain over long-horizon interaction.
- Diagnoses two coupled failure modes: trajectory-indexed utilities disperse limited feedback across an expanding state space, and trajectory-level rewards jointly credited to co-retrieved memories push irrelevant experiences into a "memory-reward trap."
- Proposes a reduced-order parameterization: a fixed set of semantic coordinates factorized by outcome polarity and memory dynamics, whose contents are updated or replaced rather than appended, bounding the utility support.
- Theoretical results argue the reduced-order form raises average feedback per utility coordinate and characterize steady-state occupancy of erroneous coordinates under a generic coordinate-transition model.
- On ALFWorld and LifelongAgentBench: improved task performance, Cold-Q ratio down 80.0%, feedback density up ~6.0x, maintained memory size down 84.4%, and LLM calls cut 21.1%; code released on GitHub.