2017

Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation

Sharma, Shikhar, Asri, Layla El, Schulz, Hannes et al.

Understand

Automated metrics such as BLEU are widely used in the machine translation literature.

They have also been used recently in the dialogue community for evaluating dialogue response generation.
However, previous work in dialogue response generation has shown that these metrics do not correlate strongly with human judgment in the non task-oriented dialogue setting.
Task-oriented dialogue responses are expressed on narrower domains and exhibit lower diversity.

Built on

A coefficient of agreement for nominal scales
Jacob Cohen. 1960 · 1960
Earlier work this paper cites.
How to do things with words
John Langshaw Austin. 1962 · 1962
Earlier work this paper cites.
Generating coherent argumentative paragraphs
Michael Elhadad. 1992 · 1992
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
Natural language generation in the ibm flight information system
Scott Axelrod. 2000 · 2000
Earlier work this paper cites.
Stochastic language generation for spoken dialogue systems
Alice H. Oh and Alexander I. Rudnicky. 2000 · 2000
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
Chin-Yew Lin. 2004 · 2004
Earlier work this paper cites.
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments
Satanjeev Banerjee and Alon Lavie. 2005 · 2005
Earlier work this paper cites.
Understanding interobserver agreement: the kappa statistic
Anthony J Viera, Joanne M Garrett, et al. 2005 · 2005
Earlier work this paper cites.

Similar

Individual and domain adaptation in sentence planning for dialogue
Marilyn Walker, Amanda Stent, François Mairesse, and Rashmi Prasad. 2007 · 2007
Cited alongside, same era.
Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings
Ondřej Dušek and Filip Jurcicek. 2016 · 2008
Cited alongside, same era.
Natural language generation as planning under uncertainty for spoken dialogue systems
Verena Rieser and Oliver Lemon. 2009 · 2009
Cited alongside, same era.
A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics
Vasile Rus and Mihai C. Lintean. 2012 · 2012
Cited alongside, same era.
Bootstrapping dialog systems with word embeddings
Gabriel Forgues, Joelle Pineau, Jean-Marie Larcheveque, and Real Tremblay. 2014 · 2014
Cited alongside, same era.
The second dialog state tracking challenge
Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014 · 2014
Cited alongside, same era.
Skip-thought vectors
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015 · 2015
Cited alongside, same era.
Incorporating unstructured textual knowledge sources into neural dialogue systems
Ryan Lowe, Nissan Pow, IV Serban, Laurent Charlin, and Joelle Pineau. 2015 · 2015
Cited alongside, same era.
Semantically conditioned LSTM-based natural language generation for spoken dialogue systems
Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015b · 2015
Cited alongside, same era.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015 · 2015
Cited alongside, same era.
Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking
Tsung-Hsien Wen, Milica Gašić, Dongho Kim, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015a
Cited in the paper.

Then

Adversarial evaluation of dialogue models
Original
Anjuli Kannan and Oriol Vinyals. 2016 · 2016
Later among the works it cites.
Deep reinforcement learning for dialogue generation
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016 · 2016
Later among the works it cites.
How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016 · 2016
Later among the works it cites.
Crowd-sourcing nlg data: Pictures elicit better data
Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016 · 2016
Later among the works it cites.
A hierarchical latent variable encoder-decoder model for generating dialogues
Original
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2016 · 2016
Later among the works it cites.
Frames: A corpus for adding memory to goal-oriented dialogue systems
Original
Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017 · 2017
Closest in time.
Adversarial learning for neural dialogue generation
Original
Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. 2017 · 2017
Closest in time.
Towards an automatic turing test: Learning to evaluate dialogue responses
Ryan Lowe, Michael Noseworthy, Iulian V. Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017 · 2017
Closest in time.
Natural language generation in dialogue using lexicalized and delexicalized data
Shikhar Sharma, Jing He, Kaheer Suleman, Hannes Schulz, and Philip Bachman. 2017 · 2017
Closest in time.
deltableu: A discriminative metric for generation tasks with intrinsically diverse targets
Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015 · 2073
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…