Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models

Understand

Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters.

• We propose Cross-lingual Expert Language Models (X-ELM), which mitigate this competition by independently training language models on subsets of the multilingual corpus.
• This process specializes X-ELMs to different languages while remaining effective as a multilingual ensemble.
• Our experiments show that when given the same compute budget, X-ELM outperforms jointly trained multilingual models across all considered languages and that these gains transfer to downstream tasks.