Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - A systematic empirical study of natively unified multimodal (language + visual understanding + visual generation) pretraining, mapping how modalities transfer knowledge and interact, and distilling actionable design recipes. It matters because it turns largely ad-hoc unified-model design choices into controlled, measured guidance, validated at 13.5B MoE / 2T-token scale.
- Knowledge flow: Transfer among language, visual understanding, and visual generation is disentangled and shown to be asymmetric, with distinct patterns of cross-modal influence.
- Synergy vs. competition: Data "complexity" largely decides whether modalities help or fight each other; shared attention and normalization paired with modality-specific FFN layers promote synergy, and this holds across different visual tokenizer designs.
- Early unification: Joint training of modalities from the very start beats late alignment or sequential training; delayed integration induces a "vision laziness" effect where the model leans on language priors.
- Recipes & scale: Derived pretraining recipes reach strong generative performance with only 5% of the compute budget, and the findings are validated by training multiple 13.5B MoE models on 2T tokens.
Sources (1)
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
TL;DR - A systematic empirical study of natively unified multimodal (language + visual understanding + visual generation) pretraining, mapping how modalities transfer knowledge and interact, and distilling actionable design recipes. It matters because it turns largely ad-hoc unified-model design choices into controlled, measured guidance, validated at 13.5B MoE / 2T-token scale.
- Knowledge flow: Transfer among language, visual understanding, and visual generation is disentangled and shown to be asymmetric, with distinct patterns of cross-modal influence.
- Synergy vs. competition: Data "complexity" largely decides whether modalities help or fight each other; shared attention and normalization paired with modality-specific FFN layers promote synergy, and this holds across different visual tokenizer designs.
- Early unification: Joint training of modalities from the very start beats late alignment or sequential training; delayed integration induces a "vision laziness" effect where the model leans on language priors.
- Recipes & scale: Derived pretraining recipes reach strong generative performance with only 5% of the compute budget, and the findings are validated by training multiple 13.5B MoE models on 2T tokens.