BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling

Understand

The pre-training of large language models usually requires massive amounts of resources, both in terms of computation and data.

• Frequently used web sources such as Common Crawl might contain enough noise to make this pre-training sub-optimal.
• In this work, we experiment with different sampling methods from the Spanish version of mC4, and present a novel data-centric technique which we name $\textit{perplexity sampling}$ that enables the pre-training of language models in roughly half the amount of steps and using one fifth of the data.
• The resulting models are comparable to the current state-of-the-art, and even achieve better results for certain tasks.