Generative Embedding Benchmark: How Much Information Survives in a Dense Embedding?

TL;DR - GEB (Generative Embedding Benchmark) evaluates dense embeddings by having a decoder answer VQA questions from a frozen embedding plus question text alone, measuring how much answer-relevant information actually survives compression. It matters because separability-based benchmarks can look strong while hiding severe generative information bottlenecks.

• Setup: a decoder sees only the frozen embedding and the question — no original image or intermediate visual features; dataset has an 1,800-item dev split and held-out 900-item test split spanning natural images, scene text, and visual documents.
• Seven public embedding models were tested under a common decoder/training recipe in visual-only and vision-language joint modes; visual-only test scores fall in a narrow 28.25–33.21 band.
• Joint image-question encoding lifts all five VLM-based embedding models, with the best reaching 65.56, versus 84.30 for a Qwen3-VL-2B reference that retains access to the original image.
• Controls (text-only, zero, and shuffled embeddings) score below matched embeddings, confirming signal comes from the embedding; scene text and visual-document content is far harder to recover than natural-image content.