Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages

Understand

We present Samanantar, the largest publicly available parallel corpora collection for Indic languages.

• The collection contains a total of 49.7 million sentence pairs between English and 11 Indic languages (from two language families).
• Specifically, we compile 12.4 million sentence pairs from existing, publicly-available parallel corpora, and additionally mine 37.4 million sentence pairs from the web, resulting in a 4x increase.
• We mine the parallel sentences from the web by combining many corpora, tools, and methods: (a) web-crawled monolingual corpora, (b) document OCR for extracting sentences from scanned documents, (c) multilingual representation models for aligning sentences, and (d) approximate nearest neighbor search for searching in a large collection of sentences.