2020

Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols

Finch, Sarah E., Choi, Jinho D.

Understand

As conversational AI-based dialogue management has increasingly become a trending topic, the need for a standardized and reliable evaluation procedure grows even more pressing.

The current state of affairs suggests various evaluation protocols to assess chat-oriented dialogue management systems, rendering it difficult to conduct fair comparative studies across different approaches and gain an insightful understanding of their values.
To foster this research, a more robust evaluation protocol must be set in place.
This paper presents a comprehensive synthesis of both automated and human evaluation methods on dialogue systems, identifying their shortcomings while accumulating evidence towards the most effective evaluation dimensions.

Built on

Survey on evaluation methods for dialogue systems
Original
Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2019 · 1905
Earlier work this paper cites.
Challenges in building intelligent open-domain dialog systems
Original
Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2019 · 1905
Earlier work this paper cites.
Improving Knowledge-aware Dialogue Generation via Knowledge Base Question Answering
Original
Jian Wang, Junhao Liu, Wei Bi, Xiaojiang Liu, Kejing He, Ruifeng Xu, and Min Yang. 2020 · 1912
Earlier work this paper cites.
PARADISE: A framework for evaluating spoken dialogue agents
Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997 · 1997
Earlier work this paper cites.
Evaluation metrics for language models
Stanley F Chen, Douglas Beeferman, and Roni Rosenfeld. 1998 · 1998
Earlier work this paper cites.
Towards a Human-like Open-Domain Chatbot
Original
Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020 · 2001
Earlier work this paper cites.
Bleu: a Method for Automatic Evaluation of Machine Translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
ROUGE: A Package for Automatic Evaluation of Summaries
Chin-Yew Lin. 2004 · 2004
Earlier work this paper cites.
A diversity-promoting objective function for neural conversation models
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016 · 2016
Earlier work this paper cites.
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016 · 2016
Earlier work this paper cites.
Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation
Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. 2016 · 2016
Earlier work this paper cites.
A Syntactically Constrained Bidirectional-Asynchronous Approach for Emotional Conversation Generation
Jingyuan Li and Xiao Sun. 2018 · 2018
Earlier work this paper cites.

Similar

Knowledge Diffusion for Neural Dialogue Generation
Shuman Liu, Hongshen Chen, Zhaochun Ren, Yang Feng, Qun Liu, and Dawei Yin. 2018 · 2018
Cited alongside, same era.
An Auto-Encoder Matching Model for Learning Utterance-Level Semantic Dependency in Dialogue Generation
Liangchen Luo, Jingjing Xu, Junyang Lin, Qi Zeng, and Xu Sun. 2018 · 2018
Cited alongside, same era.
Towards Exploiting Background Knowledge for Building Conversation Systems
Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018 · 2018
Cited alongside, same era.
Extending Neural Generative Conversational Model using External Knowledge Sources
Prasanna Parthasarathi and Joelle Pineau. 2018 · 2018
Cited alongside, same era.
Conversational ai: The science behind the alexa prize
Original
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, et al. 2018 · 2018
Cited alongside, same era.
Better Conversations by Modeling, Filtering, and Optimizing for Coherence and Diversity
Xinnuo Xu, Ondřej Dušek, Ioannis Konstas, and Verena Rieser. 2018 · 2018
Cited alongside, same era.
Augmenting End-to-End Dialogue Systems With Commonsense Knowledge
Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018 · 2018
Cited alongside, same era.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018 · 2018
Cited alongside, same era.
Boosting Dialog Response Generation
Wenchao Du and Alan W Black. 2019 · 2019
Cited alongside, same era.
Neural approaches to conversational ai
Jianfeng Gao, Michel Galley, Lihong Li, et al. 2019 · 2019
Cited alongside, same era.
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. 2019 · 2019
Cited alongside, same era.
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings
Sarik Ghazarian, Johnny Wei, Aram Galstyan, and Nanyun Peng. 2019 · 2019
Cited alongside, same era.

Then

Unifying human and statistical evaluation for natural language generation
Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. 2019 · 2019
Later among the works it cites.
Incremental Transformer with Deliberation Decoder for Document Grounded Conversations
Zekang Li, Cheng Niu, Fandong Meng, Yang Feng, Qian Li, and Jie Zhou. 2019 · 2019
Later among the works it cites.
MoEL: Mixture of Empathetic Listeners
Zhaojiang Lin, Andrea Madotto, Jamin Shin, Peng Xu, and Pascale Fung. 2019 · 2019
Later among the works it cites.
Personalizing Dialogue Agents via Meta-Learning
Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. 2019 · 2019
Later among the works it cites.
Multimodal Dialogue System Evaluation: A Case Study Applying Usability Standards
Andrei Malchanau, Volha Petukhova, and Harry Bunt. 2019 · 2019
Later among the works it cites.
Are Training Samples Correlated? Learning to Generate Dialogue Responses with Multiple References
Lisong Qiu, Juntao Li, Wei Bi, Dongyan Zhao, and Rui Yan. 2019 · 2019
Later among the works it cites.
Learning to Abstract for Memory-augmented Conversational Response Generation
Zhiliang Tian, Wei Bi, Xiaopeng Li, and Nevin L. Zhang. 2019 · 2019
Later among the works it cites.
Proactive Human-Machine Conversation with Explicit Conversation Goal
Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019 · 2019
Later among the works it cites.
ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation
Hainan Zhang, Yanyan Lan, Liang Pang, Jiafeng Guo, and Xueqi Cheng. 2019 · 2019
Later among the works it cites.
Unsupervised Context Rewriting for Open Domain Conversation
Kun Zhou, Kai Zhang, Yu Wu, Shujie Liu, and Jingsong Yu. 2019 · 2019
Later among the works it cites.
Retrieval-Enhanced Adversarial Training for Neural Response Generation
Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, and Ting Liu. 2019 · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…