How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders

Understand

This study explores how bilingual language models develop complex internal representations.

• We employ sparse autoencoders to analyze internal representations of bilingual language models with a focus on the effects of training steps, layers, and model sizes.
• Our analysis shows that language models first learn languages separately, and then gradually form bilingual alignments, particularly in the mid layers.
• We also found that this bilingual tendency is stronger in larger models.