Improving Lexical Choice in Neural Machine Translation

Understand

We explore two solutions to the problem of mistranslating rare words in neural machine translation.

• First, we argue that the standard output layer, which computes the inner product of a vector representing the context with all possible output word embeddings, rewards frequent words disproportionately, and we propose to fix the norms of both vectors to a constant value.
• Second, we integrate a simple lexical module which is jointly trained with the rest of the model.
• We evaluate our approaches on eight language pairs with data sizes ranging from 100k to 8M words, and achieve improvements of up to +4.3 BLEU, surpassing phrase-based translation in nearly all settings.