Douyin Multimodal Embedding Model Technical Report

TL;DR - A technical report on Douyin Multimodal Embedding (DME), a two-stage-trained MLLM embedding model that aims to match CoT-style fine-grained discrimination while serving as cheaply as a standard contrastive encoder. It matters because billion-scale search/recommendation needs both retrieval quality and inference efficiency, which existing embedding models rarely deliver together.

• Stage 1 is large-scale contrastive pre-training to build a unified multimodal embedding space spanning broad modality and task coverage; Stage 2 targets "semantic sufficiency" — embeddings grounded in retrieval-relevant evidence that preserve counterpart-side fine-grained semantics.
• Two Stage-2 mechanisms: Evidence-Grounded Typed Latent Reasoning (organizes retrieval evidence via hidden-space latent reasoning) and Cross-Conditional Reconstruction (cross-directional autoregressive reconstruction of the counterpart). Both are training-only, adding only marginal query-side overhead at serving time.
• Reports state-of-the-art MMEB-v2 scores at comparable scales: 74.8 (2B) and 78.4 (9B), with noted strength on video and visual-document tasks.
• Production results: +2.92% relative on Douyin's in-house offline eval, deployed across generative, image, and AI search scenarios, and +0.1% Lifetime (LT) in online A/B testing on Douyin search.