How much pretraining data do language models need to learn syntax?

Understand

Transformers-based pretrained language models achieve outstanding results in many well-known NLU benchmarks.

• However, while pretraining methods are very convenient, they are expensive in terms of time and resources.
• This calls for a study of the impact of pretraining data size on the knowledge of the models.
• We explore this impact on the syntactic capabilities of RoBERTa, using models trained on incremental sizes of raw text data.