Concept Space Alignment in Multilingual LLMs

Understand

Multilingual large language models (LLMs) seem to generalize somewhat across languages.

• We hypothesize this is a result of implicit vector space alignment.
• Evaluating such alignment, we see that larger models exhibit very high-quality linear alignments between corresponding concepts in different languages.
• Our experiments show that multilingual LLMs suffer from two familiar weaknesses: generalization works best for languages with similar typology, and for abstract concepts.