Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMs

TL;DR - An arXiv cs.CV paper identifying "Role-Break," a head-level signature where attention heads deviate from their faithful contextual behavior during hallucination, and using it to build a cheap linear hallucination detector for VLMs. It matters because it generalizes across hallucination patterns rather than targeting a single one, which is where prior detectors break down.

• Argues real VLM hallucinations mix multiple patterns (e.g. visual-textual imbalance), so single-pattern signals don't transfer stably across models and tasks; proposes a unified head-level view instead.
• Finds hallucination-induced deviations are systematically structured across attention heads, contextual sources, and deviation directions, and become linearly readable when head identity is preserved (not pooled away).
• Detector is lightweight: no VLM fine-tuning, feature dimension under 5,000, averaging 93.23 AUROC across six VLMs and four benchmarks.
• A small-scale intervention experiment shows detected tokens can be acted on directly in the discriminative setting; mitigation beyond that scope isn't claimed.