SwissBERT: The Multilingual Language Model for Switzerland

Understand

We present SwissBERT, a masked language model created specifically for processing Switzerland-related text.

• SwissBERT is a pre-trained model that we adapted to news articles written in the national languages of Switzerland -- German, French, Italian, and Romansh.
• We evaluate SwissBERT on natural language understanding tasks related to Switzerland and find that it tends to outperform previous models on these tasks, especially when processing contemporary news and/or Romansh Grischun.
• Since SwissBERT uses language adapters, it may be extended to Swiss German dialects in future work.