FlauBERT: Unsupervised Language Model Pre-training for French

Understand

Language models have become a key step to achieve state-of-the art results in many different Natural Language Processing (NLP) tasks.

• Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level.
• This has been widely demonstrated for English using contextualized representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019b).
• In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus.