Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale

Understand

Automatic evaluation of language generation systems is a well-studied problem in Natural Language Processing.

• While novel metrics are proposed every year, a few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation, despite their known limitations.
• This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them.
• In this paper, we urge the community for more careful consideration of how they automatically evaluate their models by demonstrating important failure cases on multiple datasets, language pairs and tasks.