FRAME: separating sampling variation from representational cause in medical imaging fairness

TL;DR - FRAME is a two-step framework for determining whether subgroup performance gaps in medical imaging reflect sampling variation or representational mechanisms. Across large-scale experiments and prior studies, it attributes substantial portions of reported fairness gaps to cohort-size effects and questions interventions focused only on removing demographic information.

• FRAME first estimates the expected performance-gap distribution under exact fairness at observed subgroup sizes, then tests the remaining gap with representation-space interventions.
• Across 702,206 images and 36 encoders, sampling variation explained a median 41% of race differences and 22% of age differences.
• Adding demographic decodability did not change the remainder, while entangling demographic group with the disease direction increased the race difference from 0.077 to 0.118.
• Across 89 differences from nine studies, the reference explained a median 25% of rate gaps and 70% of AUROC gaps; image-text pretraining improved worst-group performance by about 0.05.