How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - A mechanistic analysis of why Vision-Language Models fail on compositional visual question answering, mapping failures to specific reasoning operations and the internal transformer pathways that produce them. It matters because it moves beyond aggregate accuracy to offer targeted, cause-specific strategies for improving VLM reliability.
- Proposes an "operation-centric" framework that decomposes VLM failures by both the reasoning operation (object selection, spatial relations, attribute verification) and the internal computational pathway.
- Identifies four distinct failure modes: grounding failure, reasoning failure, attribute extraction failure, and language-prior dominance—each defined by a unique link between visual grounding strength and answer correctness.
- Uses three causal interventions across all transformer layers to show pathway dissociation: grounding failures route through the feedforward network, reasoning failures through late-layer attention, and attribute-extraction failures through answer-position feedforward computation.
- Concludes that different failure types need fundamentally different fixes, providing a principled basis for targeted VLM improvements (claims are analytical/mechanistic; no benchmark accuracy numbers are given in the abstract).
Sources (1)
How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA
TL;DR - A mechanistic analysis of why Vision-Language Models fail on compositional visual question answering, mapping failures to specific reasoning operations and the internal transformer pathways that produce them. It matters because it moves beyond aggregate accuracy to offer targeted, cause-specific strategies for improving VLM reliability.
- Proposes an "operation-centric" framework that decomposes VLM failures by both the reasoning operation (object selection, spatial relations, attribute verification) and the internal computational pathway.
- Identifies four distinct failure modes: grounding failure, reasoning failure, attribute extraction failure, and language-prior dominance—each defined by a unique link between visual grounding strength and answer correctness.
- Uses three causal interventions across all transformer layers to show pathway dissociation: grounding failures route through the feedforward network, reasoning failures through late-layer attention, and attribute-extraction failures through answer-position feedforward computation.
- Concludes that different failure types need fundamentally different fixes, providing a principled basis for targeted VLM improvements (claims are analytical/mechanistic; no benchmark accuracy numbers are given in the abstract).