The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation

Understand

Does neural machine translation yield translations that are congenial with common sense? In this paper, we present a test suite to evaluate the commonsense reasoning capability of neural machine translation.

• The test suite consists of three test sets, covering lexical and contextless/contextual syntactic ambiguity that requires commonsense knowledge to resolve.
• We manually create 1,200 triples, each of which contain a source sentence and two contrastive translations, involving 7 different common sense types.
• Language models pretrained on large-scale corpora, such as BERT, GPT-2, achieve a commonsense reasoning accuracy of lower than 72% on target translations of this test suite.