Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
Ranking
Overall
79
Content
80
Popularity
77
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv cs.CV preprint that reframes LVLM object hallucination as a problem of what the model attends to rather than how much, using Logit Lens to decode high-attention visual regions and then applying training-free fixes. It matters because it offers a diagnostic signal plus targeted mitigation without retraining.
- Counter-evidence to the prevailing "insufficient visual attention" explanation: real and hallucinated objects receive equally strong visual attention in mid-to-late layers.
- Logit Lens decoding of high-attention regions separates the two cases — real-object regions decode to the target object tokens, hallucinated ones do not.
- Two identified mechanisms: visual uncertainty (confusable/semantically similar regions; masking removes the hallucination) and contextual prior (co-occurrence-driven; hallucination persists after masking and attention drifts elsewhere).
- Proposed training-free Detect-Mitigate framework: a Logit-Lens Consistency Check for detection, plus HARM (High-Attention Regions Masking) and VEED (Visual Evidence Enhanced Decoding), reported as state-of-the-art on multiple hallucination benchmarks; code promised but not yet released.
Sources (1)
Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
Public signals
Semantic Scholar citations 3 · Semantic Scholar influential citations 0
TL;DR - An arXiv cs.CV preprint that reframes LVLM object hallucination as a problem of what the model attends to rather than how much, using Logit Lens to decode high-attention visual regions and then applying training-free fixes. It matters because it offers a diagnostic signal plus targeted mitigation without retraining.
- Counter-evidence to the prevailing "insufficient visual attention" explanation: real and hallucinated objects receive equally strong visual attention in mid-to-late layers.
- Logit Lens decoding of high-attention regions separates the two cases — real-object regions decode to the target object tokens, hallucinated ones do not.
- Two identified mechanisms: visual uncertainty (confusable/semantically similar regions; masking removes the hallucination) and contextual prior (co-occurrence-driven; hallucination persists after masking and attention drifts elsewhere).
- Proposed training-free Detect-Mitigate framework: a Logit-Lens Consistency Check for detection, plus HARM (High-Attention Regions Masking) and VEED (Visual Evidence Enhanced Decoding), reported as state-of-the-art on multiple hallucination benchmarks; code promised but not yet released.