Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv cs.CV preprint arguing that gains in "thinking with images" vision-language models come from the structured text emitted at tool-call time (tool name, coordinates, target, intent), not from the returned pixels. It matters because it questions a core assumption of tool-augmented visual reasoning and offers a cheaper, faster alternative.
- Introduces TextCall ("call-but-no-return"): keeps the tool-call scaffold but replaces returned images with the placeholder
[Image output skipped]. - Across LoRA, full fine-tuning, and RL, TextCall matches or exceeds full thinking-with-images; under RL it preserves tool use at the reported checkpoint, avoiding a failure mode where seeing returned images makes the model stop calling tools.
- Scaffold-only input yields equivalent accuracy on matched training queries; decomposition shows both reasoning text and spatial code contribute, with the dominant component varying by task.
- Practical payoff: 29–46% latency reduction and no tool-execution API calls; authors note claims hold for current benchmarks and that building genuinely pixel-dependent tasks remains open.
Sources (1)
Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning
TL;DR - An arXiv cs.CV preprint arguing that gains in "thinking with images" vision-language models come from the structured text emitted at tool-call time (tool name, coordinates, target, intent), not from the returned pixels. It matters because it questions a core assumption of tool-augmented visual reasoning and offers a cheaper, faster alternative.
- Introduces TextCall ("call-but-no-return"): keeps the tool-call scaffold but replaces returned images with the placeholder
[Image output skipped]. - Across LoRA, full fine-tuning, and RL, TextCall matches or exceeds full thinking-with-images; under RL it preserves tool use at the reported checkpoint, avoiding a failure mode where seeing returned images makes the model stop calling tools.
- Scaffold-only input yields equivalent accuracy on matched training queries; decomposition shows both reasoning text and spatial code contribute, with the dominant component varying by task.
- Practical payoff: 29–46% latency reduction and no tool-execution API calls; authors note claims hold for current benchmarks and that building genuinely pixel-dependent tasks remains open.