Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models

TL;DR - Cross-modal attention in vision-language models acts as a question-conditioned frequency filter over image patches, helping explain why verbose prompts improve robustness while fine-grained questions increase sensitivity to image corruption. Simple prompt padding also produces measurable accuracy gains under corruption.

• Verbose rephrasings broaden the attention filter’s frequency support, reducing answer drift under corrupted images.
• Fine-grained or semantically complex questions concentrate attention on fewer visual scales, making models more fragile when corruption overlaps those frequencies.
• Across GQA and CLEVR, verbose paraphrasing reduced drift variance by 70–81% for 8B Qwen3-VL and LLaVA-OneVision models.
• The findings suggest a practical robustness technique: pad prompts with instructions such as “Please look carefully and answer.”