Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages

Understand

LLMs have become a go-to solution not just for text generation, but also for natural language understanding (NLU) tasks.

• Acquiring extensive knowledge through language modeling on web-scale corpora, they excel on English NLU, yet struggle to extend their NLU capabilities to underrepresented languages.
• In contrast, machine translation models (MT) produce excellent multilingual representations, resulting in strong translation performance even for low-resource languages.
• MT encoders, however, lack the knowledge necessary for comprehensive NLU that LLMs obtain through language modeling training on immense corpora.