MarIA: Spanish Language Models

Understand

This work presents MarIA, a family of Spanish language models and associated resources made available to the industry and the research community.

• Currently, MarIA includes RoBERTa-base, RoBERTa-large, GPT2 and GPT2-large Spanish language models, which can arguably be presented as the largest and most proficient language models in Spanish.
• The models were pretrained using a massive corpus of 570GB of clean and deduplicated texts with 135 billion words extracted from the Spanish Web Archive crawled by the National Library of Spain between 2009 and 2019.
• We assessed the performance of the models with nine existing evaluation datasets and with a novel extractive Question Answering dataset created ex novo.