What the [MASK]? Making Sense of Language-Specific BERT Models

Understand

Recently, Natural Language Processing (NLP) has witnessed an impressive progress in many areas, due to the advent of novel, pretrained contextual representation models.

• In particular, Devlin et al.
• (2019) proposed a model, called BERT (Bidirectional Encoder Representations from Transformers), which enables researchers to obtain state-of-the art performance on numerous NLP tasks by fine-tuning the representations on their data set and task, without the need for developing and training highly-specific architectures.
• The authors also released multilingual BERT (mBERT), a model trained on a corpus of 104 languages, which can serve as a universal language model.