CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

Understand

Pre-training text representations have led to significant improvements in many areas of natural language processing.

• The quality of these models benefits greatly from the size of the pretraining corpora as long as its quality is preserved.
• In this paper, we describe an automatic pipeline to extract massive high-quality monolingual datasets from Common Crawl for a variety of languages.
• Our pipeline follows the data processing introduced in fastText (Mikolov et al., 2017; Grave et al., 2018), that deduplicates documents and identifies their language.