Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

Understand

Cross-lingual alignment in multilingual language models is typically attributed to joint training: shared parameters, mixed-language batches, or explicit alignment objectives.

• We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training.
• By testing on strictly monolingual language models, such as the Goldfish model families and independently developed models from different research labs, we find three results.
• Correlation: these models develop alignable representational geometry across layers, with alignment strengthening as data scale, model scale, or linguistic proximity increases.