How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs

Understand

Analysing translation quality in regards to specific linguistic phenomena has historically been difficult and time-consuming.

• Neural machine translation has the attractive property that it can produce scores for arbitrary translations, and we propose a novel method to assess how well NMT systems model specific linguistic phenomena such as agreement over long distances, the production of novel words, and the faithful translation of polarity.
• The core idea is that we measure whether a reference translation is more probable under a NMT model than a contrastive translation which introduces a specific type of error.
• We present LingEval97, a large-scale data set of 97000 contrastive translation pairs based on the WMT English->German translation task, with errors automatically created with simple rules.