Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings

Understand

Machine translation is highly sensitive to the size and quality of the training data, which has led to an increasing interest in collecting and filtering large parallel corpora.

• In this paper, we propose a new method for this task based on multilingual sentence embeddings.
• In contrast to previous approaches, which rely on nearest neighbor retrieval with a hard threshold over cosine similarity, our proposed method accounts for the scale inconsistencies of this measure, considering the margin between a given sentence pair and its closest candidates instead.
• Our experiments show large improvements over existing methods.