GottBERT: a pure German Language Model

Understand

Pre-trained language models have significantly advanced natural language processing (NLP), especially with the introduction of BERT and its optimized version, RoBERTa.

• While initial research focused on English, single-language models can be advantageous compared to multilingual ones in terms of pre-training effort, overall resource efficiency or downstream task performance.
• Despite the growing popularity of prompt-based LLMs, more compute-efficient BERT-like models remain highly relevant.
• In this work, we present the first German single-language RoBERTa model, GottBERT, pre-trained exclusively on the German portion of the OSCAR dataset.