On the Cross-lingual Transferability of Monolingual Representations

Understand

State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting.

• This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions.
• We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level.
• More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective, freezing parameters of all other layers.