Do Medical Vision Models Reason About Anatomy? Probing the Spatial Inductive Biases of Learned Visual Representations

TL;DR - SPAR-Bench tests whether medical vision models can reason spatially about abdominal CT anatomy, finding that they mostly encode typical organ locations rather than compare structures within individual patients. This exposes a key limitation hidden by diagnostic accuracy and standard probing methods.

• Eight probes separate coordinate localization, relational reasoning, and spatial queries across multi-organ abdominal CT scans.
• Within-slice comparison tasks remained near chance across five architectures and three medical foundation models, despite scaling and finetuning.
• Apparent in-domain success vanished under zero-shot transfer, suggesting memorization of canonical anatomy rather than image-based spatial computation.
• Using full token features instead of pooled representations raised relational recovery from 0.7% to 67.8%, showing pooled probes can substantially underestimate encoded information.