Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

TL;DR - Independently trained monolingual language models develop cross-lingually alignable internal representations without shared training data or explicit alignment objectives. This suggests language structure itself may enable modular multilingual systems assembled from monolingual models.

• Alignment strengthens with greater data and model scale, as well as closer linguistic similarity.
• A single Procrustes rotation learned from parallel sentences can map hidden states between models.
• Rotated English residual states patched into a German model transferred factual content, often changing its cloze prediction to the English donor model’s answer.
• The findings point toward model stitching, merging, and modular multilingual architectures.