BLEU might be Guilty but References are not Innocent

Understand

The quality of automatic metrics for machine translation has been increasingly called into question, especially for high-quality systems.

• This paper demonstrates that, while choice of metric is important, the nature of the references is also critical.
• We study different methods to collect references and compare their value in automated evaluation by reporting correlation with human evaluation for a variety of systems and metrics.
• Motivated by the finding that typical references exhibit poor diversity, concentrating around translationese language, we develop a paraphrasing task for linguists to perform on existing reference translations, which counteracts this bias.