Identifying Necessary Elements for BERT's Multilinguality

Understand

It has been shown that multilingual BERT (mBERT) yields high quality multilingual representations and enables effective zero-shot transfer.

• This is surprising given that mBERT does not use any crosslingual signal during training.
• While recent literature has studied this phenomenon, the reasons for the multilinguality are still somewhat obscure.
• We aim to identify architectural properties of BERT and linguistic properties of languages that are necessary for BERT to become multilingual.