Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models

TL;DR - Medical-Checklist is a binary caption-selection benchmark testing whether multimodal models genuinely comprehend medical images. Its evaluation suggests strong performance on tasks such as Med-VQA may not reflect reliable visual understanding needed for clinical use.

• Each example pairs an image with correct and incorrect captions differing by one substituted medical concept.
• The simple format supports unified evaluation across models built and trained using different approaches.
• The benchmark covers multiple medical subdomains, reduces potential data biases, and tests out-of-distribution inputs.
• Four state-of-the-art medical multimodal models showed image-comprehension weaknesses despite strong task-specific performance.