Symbal: Detecting Systematic Misalignments in Model-Generated Captions

TL;DR - Symbal is a method (with a companion benchmark, SymbalBench) for detecting "systematic misalignments"—recurring caption errors tied to specific visual features—in MLLM-generated image captions, helping audit vision-language datasets without needing access to the source model.

• Defines a new task, systematic misalignment detection: finding recurring caption errors that correlate with a specific visual feature in the paired image.
• Symbal uses a dual-stage pipeline of off-the-shelf foundation models to identify misalignments and summarize them in natural language.
• SymbalBench provides 1.7M image-text pairs across natural and medical domains, organized into 420 datasets with annotated misalignments.
• Reports correctly identifying misalignments in 63.8% of datasets (~4x over the closest baseline), plus real-world results auditing captions from four MLLMs; code is open-sourced.