FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification

TL;DR - FaithEyes is a multi-agent framework that trains vision-language models to use image-processing tools only when their outputs genuinely aid reasoning. It improves tool faithfulness while maintaining competitive or superior benchmark accuracy.

• A VLM judges whether each cropped or manipulated process image helps answer the question.
• Helpfulness judgments guide subsequent reasoning and scale rewards to discourage decorative or misaligned tool calls.
• At inference, the model acts as its own judging subagent, avoiding reliance on an external evaluator.
• Training uses a two-stage supervised fine-tuning and reinforcement learning pipeline on adapted open-source data.