Selective State-Space Adaptation and Retrieval for Language Model Reasoning
Merged summary
TL;DR - MaLoRA and MaRA add selective state-space recurrence to frozen language-model adapters, enabling input-dependent token adaptation and retrieval across context segments. They consistently outperform standard LoRA on multi-hop reasoning benchmarks.
- MaLoRA dynamically modulates adapter scaling using recurrent token-level state.
- MaRA maintains cross-segment state and selects query-relevant context before generation.
- Across three 7B–9B backbones and two benchmarks, the methods improve average F1 by 6.8 points over LoRA.
- Gains reach 9.3 F1 on the hardest setting and transfer to long-context RULER QA-2 tests.
Sources (1)
Selective State-Space Adaptation and Retrieval for Language Model Reasoning
TL;DR - MaLoRA and MaRA add selective state-space recurrence to frozen language-model adapters, enabling input-dependent token adaptation and retrieval across context segments. They consistently outperform standard LoRA on multi-hop reasoning benchmarks.
- MaLoRA dynamically modulates adapter scaling using recurrent token-level state.
- MaRA maintains cross-segment state and selects query-relevant context before generation.
- Across three 7B–9B backbones and two benchmarks, the methods improve average F1 by 6.8 points over LoRA.
- Gains reach 9.3 F1 on the hardest setting and transfer to long-context RULER QA-2 tests.