Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

TL;DR - This paper identifies semantic specialization among Vision Transformer attention heads and uses it to design Ariadne Attention. The hybrid mechanism matches full attention across 22 image and video tasks while using 6.5× less attention compute.

• ViT heads differentiate into object and background specialists, especially under full attention.
• The proposed SHS-Index quantifies this specialization and strongly tracks downstream benchmark performance.
• Window interaction, token serialization, and local softmax allocation emerge as key structural drivers of specialization.
• Ariadne Attention applies these principles to achieve full-attention-level results with substantially lower compute.