Towards Explainable Evaluation Metrics for Natural Language Generation

Understand

Unlike classical lexical overlap metrics such as BLEU, most current evaluation metrics (such as BERTScore or MoverScore) are based on black-box language models such as BERT or XLM-R.

• They often achieve strong correlations with human judgments, but recent research indicates that the lower-quality classical metrics remain dominant, one of the potential reasons being that their decision processes are transparent.
• To foster more widespread acceptance of the novel high-quality metrics, explainability thus becomes crucial.
• In this concept paper, we identify key properties and propose key goals of explainable machine translation evaluation metrics.