HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models

TL;DR - HAFI-VLM diagnoses "spectral response rigidity" in pretrained vision encoders — fixed layerwise frequency profiles that barely shift under fine-tuning — and fixes it with a task-conditioned frequency pathway that improves fine-grained visual perception in VLMs.

• Core diagnosis: because vision encoders see only images (no query), they cannot adapt spectral extraction to the evidence a given task needs, producing persistent encoder-specific spectral profiles.
• Hierarchical Adaptive Frequency Injection (HAFI) pulls complementary low-, mid-, and high-frequency evidence at multiple encoder depths via text-modulated, spatially aligned cross-attention, while preserving pretrained semantic representations.
• A Visual Enrichment Layer Adapter recalibrates shallow LLM attention so the enriched visual tokens are actually used.
• On LLaVA-1.5 and Qwen2.5-VL, it reports consistent gains in general VQA, text-rich understanding, and hallucination robustness — beating representation-level methods and most resolution/cropping approaches without extra high-resolution encoding; mechanistic analysis shows restored task-dependent spectral allocation.