CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Understand

Language identification (LID) is a fundamental step in curating multilingual corpora.

• However, LID models still perform poorly for many languages, especially on the noisy and heterogeneous web data often used to train multilingual language models.
• In this paper, we introduce CommonLID, a community-driven, human-annotated LID benchmark for the web domain, covering 109 languages.
• Many of the included languages have been previously under-served, making CommonLID a key resource for developing more representative high-quality text corpora.