Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval

TL;DR - UniME-R1 is an embedder-adviser framework for unified multimodal retrieval that generates Retrieval-Centric Chain-of-Thought (RC-CoT) conditioned on actual retrieval feedback rather than on the query alone. It matters because it targets the specific failure mode of LVLM retrievers — confusing semantically similar candidates — by reasoning about what the retriever got wrong.

• Reframes CoT for retrieval: instead of explaining the query, an adviser inspects initially retrieved candidates individually to surface the discriminative cues the embedder confused.
• Two-mode operation: if the target is in the initial top-k, it reranks directly; otherwise it emits RC-CoT to redirect the search and performs full-corpus re-retrieval with a dual-mode embedder.
• Training mines hard negatives to simulate realistic retrieval failures, jointly optimizes direct and RC-CoT-augmented retrieval, and aligns the adviser to retrieval outcomes via supervised learning plus retrieval-oriented RL.
• Evaluated on MMEB-V2 and additional general multimodal retrieval benchmarks, reported as consistently improving over strong baselines (no specific numbers given in the abstract).