RUMBA: Russian User Memory Benchmark
Merged summary
TL;DR - RUMBA is a bilingual benchmark centered on Russian for diagnosing long-term conversational memory in LLMs. It evaluates retrieval and reasoning across timestamped sessions rather than relying only on aggregate retrieval metrics.
- Provides a taxonomy spanning semantic type, session scope, temporal reasoning, and temporal-expression explicitness.
- Uses QA pairs requiring information retrieval, combination, and reasoning across conversations.
- Includes an aligned English subset based on the same methodology.
- Evaluates memory systems and long-context models across benchmark slices to expose their strengths and failure modes.
Sources (1)
RUMBA: Russian User Memory Benchmark
TL;DR - RUMBA is a bilingual benchmark centered on Russian for diagnosing long-term conversational memory in LLMs. It evaluates retrieval and reasoning across timestamped sessions rather than relying only on aggregate retrieval metrics.
- Provides a taxonomy spanning semantic type, session scope, temporal reasoning, and temporal-expression explicitness.
- Uses QA pairs requiring information retrieval, combination, and reasoning across conversations.
- Includes an aligned English subset based on the same methodology.
- Evaluates memory systems and long-context models across benchmark slices to expose their strengths and failure modes.