How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

Understand

We investigate evaluation metrics for dialogue response generation systems where supervised labels, such as task completion, are not available.

• Recent works in response generation have adopted metrics from machine translation to compare a model's generated response to a single target response.
• We show that these metrics correlate very weakly with human judgements in the non-technical Twitter domain, and not at all in the technical Ubuntu domain.
• We provide quantitative and qualitative results highlighting specific weaknesses in existing metrics, and provide recommendations for future development of better automatic evaluation metrics for dialogue systems.