Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information

TL;DR - The paper identifies Visual Retrieval Heads (VRHs), a small subset of attention heads that enable vision-language models to connect text descriptions to relevant image regions. Their causal importance and transfer across tasks and related architectures reveal a shared, sparse mechanism for visual grounding.

• VRHs comprise roughly 1.7–2.6% of attention heads across 11 VLMs and five referring-expression benchmarks.
• Masking the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, whereas masking random heads has little effect.
• Heads discovered through bounding-box prediction remain causal for attribute, spatial, counting, and visual-math tasks.
• VRHs preserve output formatting while disrupting localization and transfer between VLMs sharing an LLM backbone despite differences in their visual components and instruction tuning.