Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus

Understand

Large text corpora are increasingly important for a wide variety of Natural Language Processing (NLP) tasks, and automatic language identification (LangID) is a core technology needed to collect such datasets in a multilingual context.

• LangID is largely treated as solved in the literature, with models reported that achieve over 90% average F1 on as many as 1,366 languages.
• We train LangID models on up to 1,629 languages with comparable quality on held-out test sets, but find that human-judged LangID accuracy for web-crawl text corpora created using these models is only around 5% for many lower-resource languages, suggesting a need for more robust evaluation.
• Further analysis revealed a variety of error modes, arising from domain mismatch, class imbalance, language similarity, and insufficiently expressive models.