RUMBA: Russian User Memory Benchmark

TL;DR - RUMBA is a bilingual benchmark centered on Russian for diagnosing long-term conversational memory in LLMs. It evaluates retrieval and reasoning across timestamped sessions rather than relying only on aggregate retrieval metrics.

• Provides a taxonomy spanning semantic type, session scope, temporal reasoning, and temporal-expression explicitness.
• Uses QA pairs requiring information retrieval, combination, and reasoning across conversations.
• Includes an aligned English subset based on the same methodology.
• Evaluates memory systems and long-context models across benchmark slices to expose their strengths and failure modes.