Can Automatic Metrics Assess High-Quality Translations?

Understand

Automatic metrics for evaluating translation quality are typically validated by measuring how well they correlate with human assessments.

• However, correlation methods tend to capture only the ability of metrics to differentiate between good and bad source-translation pairs, overlooking their reliability in distinguishing alternative translations for the same source.
• In this paper, we confirm that this is indeed the case by showing that current metrics are insensitive to nuanced differences in translation quality.
• This effect is most pronounced when the quality is high and the variance among alternatives is low.