Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
Ranking
Overall
78
Content
95
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - The paper identifies Visual Retrieval Heads (VRHs), a small subset of attention heads that enable vision-language models to connect text descriptions to relevant image regions. Their causal importance and transfer across tasks and related architectures reveal a shared, sparse mechanism for visual grounding.
- VRHs comprise roughly 1.7–2.6% of attention heads across 11 VLMs and five referring-expression benchmarks.
- Masking the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, whereas masking random heads has little effect.
- Heads discovered through bounding-box prediction remain causal for attribute, spatial, counting, and visual-math tasks.
- VRHs preserve output formatting while disrupting localization and transfer between VLMs sharing an LLM backbone despite differences in their visual components and instruction tuning.
Sources (1)
Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - The paper identifies Visual Retrieval Heads (VRHs), a small subset of attention heads that enable vision-language models to connect text descriptions to relevant image regions. Their causal importance and transfer across tasks and related architectures reveal a shared, sparse mechanism for visual grounding.
- VRHs comprise roughly 1.7–2.6% of attention heads across 11 VLMs and five referring-expression benchmarks.
- Masking the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, whereas masking random heads has little effect.
- Heads discovered through bounding-box prediction remain causal for attribute, spatial, counting, and visual-math tasks.
- VRHs preserve output formatting while disrupting localization and transfer between VLMs sharing an LLM backbone despite differences in their visual components and instruction tuning.