MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer

Understand

The main goal behind state-of-the-art pre-trained multilingual models such as multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer.

• However, due to limited model capacity, their transfer performance is the weakest exactly on such low-resource languages and languages unseen during pre-training.
• We propose MAD-X, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations.
• In addition, we introduce a novel invertible adapter architecture and a strong baseline method for adapting a pre-trained multilingual model to a new language.