RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States

TL;DR - RoMeRL is a reinforcement-learning memory framework for self-evolving LLM agents that replaces an ever-growing trajectory-indexed utility space with a fixed-dimensional per-task memory state, concentrating sparse feedback and limiting reward contamination. It matters because it makes learned agent memory both denser in feedback and cheaper to maintain over long-horizon interaction.

• Diagnoses two coupled failure modes: trajectory-indexed utilities disperse limited feedback across an expanding state space, and trajectory-level rewards jointly credited to co-retrieved memories push irrelevant experiences into a "memory-reward trap."
• Proposes a reduced-order parameterization: a fixed set of semantic coordinates factorized by outcome polarity and memory dynamics, whose contents are updated or replaced rather than appended, bounding the utility support.
• Theoretical results argue the reduced-order form raises average feedback per utility coordinate and characterize steady-state occupancy of erroneous coordinates under a generic coordinate-transition model.
• On ALFWorld and LifelongAgentBench: improved task performance, Cold-Q ratio down 80.0%, feedback density up ~6.0x, maintained memory size down 84.4%, and LLM calls cut 21.1%; code released on GitHub.