Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modeling
Ranking
Overall
67
Content
80
Popularity
36
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv preprint proposing Hierarchical Memory Mamba (HMM), which bolts a brain-inspired working/long-term memory hierarchy onto a pre-trained Mamba backbone to fix the fixed-capacity recurrent state bottleneck that hurts linear-attention models on long sequences.
- Treats the Mamba backbone's hidden states as fast "sensory memory," adds a lightweight working memory that extracts slow paragraph-level semantics (PLS), then compresses PLS into a persistent long-term memory for task-relevant retrieval.
- Reported gains over strong Mamba-based baselines: +34.3–37.1% retrieval success on Passkey Retrieval and +1.6–14.2% reasoning accuracy on LongBench-E.
- Costs only ~2% extra parameters with minimal training overhead, since it builds on an existing pre-trained backbone rather than training from scratch.
- Authors claim cross-task generalization via parametric learning, which they say is absent in other long-context Mamba variants.
Sources (1)
Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modeling
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - An arXiv preprint proposing Hierarchical Memory Mamba (HMM), which bolts a brain-inspired working/long-term memory hierarchy onto a pre-trained Mamba backbone to fix the fixed-capacity recurrent state bottleneck that hurts linear-attention models on long sequences.
- Treats the Mamba backbone's hidden states as fast "sensory memory," adds a lightweight working memory that extracts slow paragraph-level semantics (PLS), then compresses PLS into a persistent long-term memory for task-relevant retrieval.
- Reported gains over strong Mamba-based baselines: +34.3–37.1% retrieval success on Passkey Retrieval and +1.6–14.2% reasoning accuracy on LongBench-E.
- Costs only ~2% extra parameters with minimal training overhead, since it builds on an existing pre-trained backbone rather than training from scratch.
- Authors claim cross-task generalization via parametric learning, which they say is absent in other long-context Mamba variants.