Scaling Vision-Language Models Is Not Enough to Mitigate Bias

TL;DR - A study of 194 vision-language models finds that increasing model scale alone does little to mitigate complex biases. Training-data quality is more consistently associated with better worst-group performance.

• Scale-performance correlation drops from ρ=0.68 on ImageNet to ρ=0.48 on CelebA and ρ=0.05 on UrbanCars.
• Curated training data improves worst-group accuracy by up to 25% over similarly scaled uncurated data.
• Architectural effects, including patch size and image resolution, vary by bias type, benchmark, and spatial distribution.