The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

TL;DR - A causal audit of the "thinking-with-images" paradigm finds that when multimodal LLMs invoke visual tools like crop-and-zoom, the returned visual evidence often has no causal effect on the final answer, meaning reported accuracy gains largely do not come from actually looking.

• Formalizes visual tool-use as a causal graph separating observation-mediated paths from action-induced shortcuts, then intervenes at three levels: policy (tool-use vs. direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually swapping one observation under a fixed prefix).
• Introduces Visual Evidence Gain, a step-level estimand isolating each returned observation's contribution to the answer.
• Across six models and five fine-grained perception benchmarks, identifies two policy miscalibration failure modes: "Calling Without Looking" (observations have no causal effect) and "Looking Without Planning" (informative observations but incoherent call schedule).
• Trajectory-level diagnostics show aggregate accuracy gains concentrate in a small "Calibrated" minority of rollouts — the authors' "illusion of visual tool-use." Code released at OpenCausaLab/CauAudit.