The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
Ranking
Overall
78
Content
80
Popularity
75
Observed public metrics from 1 member.
Merged summary
TL;DR - A causal audit of the "thinking-with-images" paradigm finds that when multimodal LLMs invoke visual tools like crop-and-zoom, the returned visual evidence often has no causal effect on the final answer, meaning reported accuracy gains largely do not come from actually looking.
- Formalizes visual tool-use as a causal graph separating observation-mediated paths from action-induced shortcuts, then intervenes at three levels: policy (tool-use vs. direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually swapping one observation under a fixed prefix).
- Introduces Visual Evidence Gain, a step-level estimand isolating each returned observation's contribution to the answer.
- Across six models and five fine-grained perception benchmarks, identifies two policy miscalibration failure modes: "Calling Without Looking" (observations have no causal effect) and "Looking Without Planning" (informative observations but incoherent call schedule).
- Trajectory-level diagnostics show aggregate accuracy gains concentrate in a small "Calibrated" minority of rollouts — the authors' "illusion of visual tool-use." Code released at OpenCausaLab/CauAudit.
Sources (1)
The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
Public signals
Hugging Face upvotes 8 · Semantic Scholar citations 1 · Semantic Scholar influential citations 1
TL;DR - A causal audit of the "thinking-with-images" paradigm finds that when multimodal LLMs invoke visual tools like crop-and-zoom, the returned visual evidence often has no causal effect on the final answer, meaning reported accuracy gains largely do not come from actually looking.
- Formalizes visual tool-use as a causal graph separating observation-mediated paths from action-induced shortcuts, then intervenes at three levels: policy (tool-use vs. direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually swapping one observation under a fixed prefix).
- Introduces Visual Evidence Gain, a step-level estimand isolating each returned observation's contribution to the answer.
- Across six models and five fine-grained perception benchmarks, identifies two policy miscalibration failure modes: "Calling Without Looking" (observations have no causal effect) and "Looking Without Planning" (informative observations but incoherent call schedule).
- Trajectory-level diagnostics show aggregate accuracy gains concentrate in a small "Calibrated" minority of rollouts — the authors' "illusion of visual tool-use." Code released at OpenCausaLab/CauAudit.