2021

Modeling Performance in Open-Domain Dialogue with PARADISE

Walker, Marilyn, Harmon, Colin, Graupera, James et al.

Understand

There has recently been an explosion of work on spoken dialogue systems, along with an increased interest in open-domain systems that engage in casual conversations on popular topics such as movies, books and music.

These systems aim to socially engage, entertain, and even empathize with their users.
Since the achievement of such social goals is hard to measure, recent research has used dialogue length or human ratings as evaluation metrics, and developed methods for automatically calculating novel metrics, such as coherence, consistency, relevance and engagement.
Here we develop a PARADISE model for predicting the performance of Athena, a dialogue system that has participated in thousands of conversations with real users, while competing as a finalist in the Alexa Prize.

Built on

PARADISE: A general framework for evaluating spoken dialogue agents
M. A. Walker, D. Litman, C. A. Kamm, and A. Abella · 1997
Earlier work this paper cites.
Automatic prediction of problematic human-computer dialogues in How May I Help You?
I. Langkilde, M. A. Walker, J. Wright, A. Gorin, and D. Litman · 1999
Earlier work this paper cites.
Predicting automatic speech recognition performance using prosodic cues
D. J. Litman, J. B. Hirschberg, and M. Swerts · 2000
Earlier work this paper cites.
Empirical evaluation of a reinforcement learning spoken dialogue system
S. Singh, M S. Kearns, D J. Litman, and M A. Walker · 2000
Earlier work this paper cites.
Using natural language processing and discourse features to identify understanding errors in a spoken dialogue system
M. Walker, J. Wright, and I. Langkilde · 2000
Earlier work this paper cites.
Towards developing general models of usability with PARADISE
M. A. Walker, C. A. Kamm, and D. J. Litman · 2000
Earlier work this paper cites.
An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email
M. A. Walker · 2000
Earlier work this paper cites.
DATE: A dialogue act tagging scheme for evaluation
M. Walker and R. Passonneau · 2001
Earlier work this paper cites.
What’s the trouble: automatically identifying problematic dialogues in DARPA communicator dialogue systems
H. Wright Hastie, R. Prasad, and M. Walker · 2002
Earlier work this paper cites.
DARPA communicator: Cross-system results for the 2001 evaluation
M. Walker, A. Rudnicky, R. Prasad, J. Aberdeen, E. Bratt, J. Garofolo, H. Hastie, A. Le, B. Pellom, A. Potamianos, R. Passonneau, S. Roukos, G. Sanders, S. Seneff, and D. Stallard · 2002
Earlier work this paper cites.
Automatically Training a Problematic Dialogue Predictor for a Spoken Dialogue System
M. A. Walker, I. Langkilde-Geary, H. Wright Hastie, J. Wright, and A. Gorin · 2002
Earlier work this paper cites.
Automatically training a problematic dialogue predictor for a spoken dialogue system
M.A. Walker, I. Langkilde-Geary, H.W. Hastie, J.H. Wright, A.L. Gorin, et al · 2002
Earlier work this paper cites.
Effects of self-disclosure and empathy in human-computer dialogue. IEEE Spoken Language Technology Workshop. 109–112. 2008
R Higashinaka, K. Dohsaka, and H Isozaki, · 2008
Earlier work this paper cites.
Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk
C. Callison-Burch · 2009
Earlier work this paper cites.
Lexical and syntactic adaptation and their impact in deployed spoken dialog systems
S. Stoyanchev and A. Stent · 2009
Earlier work this paper cites.
Scikit-learn: Machine learning in Python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay · 2011
Earlier work this paper cites.
Data-driven response generation in social media
A. Ritter, C. Cherry, and W. B Dolan · 2011
Earlier work this paper cites.
Localized detection of speech recognition errors
S. Stoyanchev, P. Salletmayr, Jingbo Yang, and Julia Hirschberg · 2012
Earlier work this paper cites.
An annotated corpus of film dialogue for learning and characterizing character style
M. A Walker, G. I Lin, and J. Sawyer · 2012
Earlier work this paper cites.
Comparing system-response retrieval models for open-domain and casual conversational agent. 2nd Workshop on Chatbots and Conversational Agent Technologies (WOCHAT@ IVA2016). 2016
F. Charras, G. Duplessis, V. Letard, A.L. Ligozat, and S Rosset, · 2016
Earlier work this paper cites.
Evaluating prerequisite qualities for learning end-to-end dialog systems
J. Dodge, A. Gane, X. Zhang, A. Bordes, S. Chopra, A. H Miller, A. Szlam, and J. Weston · 2016
Earlier work this paper cites.
Purely Corpus-based Automatic Conversation Authoring. D. Guillaume, V. Letard, A-L. Ligozat and S. Rosset,
2016
Earlier work this paper cites.

Similar

Deep Reinforcement Learning for Dialogue Generation. Proc. of EMNLP. 1192–1202. 2016
Li, J., W. Monroe, A. Ritter, D. Jurafsky, M. Galley and J. Gao · 2016
Cited alongside, same era.
How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation
CW Liu, R. Lowe, I.V. Serban, M. Noseworthy, L. Charlin, and J. Pineau · 2016
Cited alongside, same era.
Building end-to-end dialogue systems using generative hierarchical neural network models
I V Serban, A. Sordoni, Y. Bengio, A.Courville, and J. Pineau · 2016
Cited alongside, same era.
Analysis of temporal features for interaction quality estimation
S. Ultes, A. Schmitt, and W. Minker · 2016
Cited alongside, same era.
Adversarial evaluation for open-domain dialogue generation
E.Bruni and R.l Fernandez · 2017
Cited alongside, same era.
End-to-end conversation modeling track in DSTC6
Original
C. Hori and T. Hori · 2017
Cited alongside, same era.
Dailydialog: A manually labelled multi-turn dialogue dataset
Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S.i Niu · 2017
Cited alongside, same era.
Alana: Social dialogue using an ensemble model and a ranker trained on user feedback. Proc. Alexa Prize, 2017
I. Papaioannou, A. Curry, J. L Part, I. Shalyminov, X. Xu, Y.. Yu, O. Dušek, V. Rieser, and O. Lemon · 2017
Cited alongside, same era.
Domain-independent user satisfaction reward estimation for dialogue policy learning
S. Ultes, P. Budzianowski, I. Casanueva, N. Mrkšić, L. Rojas-Barahona, P-H. Su, T-H. Wang. , M. Gašić, and S. Young · 2017
Cited alongside, same era.
Attention is all you need
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N Gomez, L. Kaiser, and I. Polosukhin · 2017
Cited alongside, same era.
Slugnerds: A named entity recognition tool for open domain dialogue systems
K.K. Bowden, J. Wu, S. Oraby, A. Misra, and M.A. Walker · 2018
Cited alongside, same era.
Gunrock: Building a human-like social bot by leveraging large scale real user data
C.Y. Chen, D.Yu, W. W. , Y.M. Yang, J. Zhang, M. Zhou, K. Jesse, A. Chau, A. Bhowmick, S. Iyer, et al · 2018
Cited alongside, same era.

Then

ViGGO: A video game corpus for data-to-text generation in open-domain conversation
J. Juraska, K. K. Bowden, and M. A. Walker · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever · 2019
Later among the works it cites.
Towards empathetic open-domain conversation models: A new benchmark and dataset
H. Rashkin, E M Smith, M Li, and Y-L Boureau · 2019
Later among the works it cites.
Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators
S. Yi, R. Goel, C. Khatri, A. Cervone, T. Chung, B. Hedayatnia, A. Venkatesh, R. Gabriel, and D. Hakkani-Tur · 2019
Later among the works it cites.
Language models are few-shot learners
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, and others · 2020
Later among the works it cites.
Emora: An inquisitive social chatbot who cares for you
S E Finch, J D Finch, A. Ahmadvand, X. Dong, R. Qi, H. Sahijwani, S. Volokhin, Z. Wang, Z. Wang, J. Choi, et al · 2020
Later among the works it cites.
Further advances in open domain dialog systems in the Third Alexa Prize Socialbot Grand Challenge
R. Gabriel, Y. Liu, A. Gottardi, M. Eric, A. Khatri, A. Chadha, Qinlang Chen, B. Hedayatnia, P. Rajan, A. Binici, et al · 2020
Later among the works it cites.
Predictive engagement: An efficient metric for automatic evaluation of open-domain dialogue systems
S. Ghazarian, R. Weischedel, A. Galstyan, and N. Peng · 2020
Later among the works it cites.
Athena: Constructing dialogues dynamically with discourse constraints
V. Harrison, J. Juraska, W. Cui, L.I. Reed, K.K. Bowden, J Wu, B. Schwarzmann, A. Ebrahimi, Rishi Rajasekaran, Nikhil Varghese, et al · 2020
Later among the works it cites.
Policy-driven neural response generation for knowledge-grounded dialogue systems
B. Hedayatnia, S. Kim, Y. Liu, K. Gopalakrishnan, M. Eric, and D. Hakkani-Tur · 2020
Later among the works it cites.
Unsupervised evaluation of interactive dialog with dialogpt
S. Mehri and M. Eskenazi · 2020
Later among the works it cites.
USR: An unsupervised and reference free evaluation metric for dialog
S. Mehri and M. Eskenazi · 2020
Later among the works it cites.
Towards holistic and automatic evaluation of open-domain dialogue generation
B. Pang, E. Nijkamp, W. Han, L. Zhou, Y. Liu, and K.i Tu · 2020
Later among the works it cites.
Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations
A. Paranjape, A. See, K. Kenealy, H.J Li, A.Hardy, P. Qi, K R Sadagopan, N M Phu, D. Soylu, and C. D Manning · 2020
Later among the works it cites.
DialoGPT: Large-scale generative pre-training for conversational response generation
Y. Zhang, S. Sun, M.l Galley, Y-C. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, and W B Dolan · 2020
Later among the works it cites.
The design and implementation of Xiaoice, an empathetic social chatbot
L. Zhou, J. Gao, Di Li, and H-Y. Shum · 2020
Later among the works it cites.
Overview of the dialogue breakdown detection challenge 4, Increasing Naturalness and Flexibility in Spoken Dialogue Interaction
R. Higashinaka, L. D’Haro, B. Abu Shawar, R. Banchs, K. Funakoshi, M. Inaba, Y. Tsunomori, and T. Takahashi, and J. Sedoc · 2021
Closest in time.
Athena 2.0: Discourse and User Modeling in Open Domain Dialogue
O. Patil, L. Reed, K. K. Bowden, J. Juraska, W. Cui, V. Harrison, R. Rajasekaran, A. Ramirez, C. Li, E. Zamora, P. Lee, J. Bheemanpally, R. Pandey, A. Ratnaparkhi, and M.A. Walker, · 2021
Closest in time.
Few shot Neural Natural Language Generation for Knowledge-graph based Response Generators
L.I. Reed and M.A. Walker · 2021
Closest in time.
Entity resolution in open-domain conversations
M Shang, T. Wang, M. Eric, J. Chen, J. Wang, M. Welch, T. Deng, A. Grewal, H. Wang, Y. Liu, D. Hakkani-Tur · 2021
Closest in time.
A comprehensive assessment of dialog evaluation metrics
Original
Y-T. Yeh, M. Eskenazi, and S. Mehri · 2021
Closest in time.
Midas: A dialog act annotation scheme for open domain humanmachine spoken conversations
D.Yu and Z. Yu · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…