Scaling Native Multimodal Pre-Training From Scratch

TL;DR - This study derives scaling laws for native vision-language pre-training under fixed compute budgets. It shows that optimal model size, token count, and data mixture can be predicted, helping allocate resources efficiently when training multimodal foundation models from scratch.

• Minimum objective loss follows a predictable compute law, while compute-optimal model size and token count follow power laws.
• Language learning is relatively insensitive to multimodal data ratios, but multimodal learning is highly sensitive to data composition.
• Text-heavy mixtures become compute-efficient at larger scales and favor allocating more compute to model capacity.
• Native multimodal pre-training improves text-only spatial reasoning and supports robust multimodal in-context learning.