2019

Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation

Garbacea, Cristina, Carton, Samuel, Yan, Shiyan et al.

Understand

We conduct a large-scale, systematic study to evaluate the existing evaluation methods for natural language generation in the context of generating online product reviews.

We compare human-based evaluators with a variety of automated evaluation procedures, including discriminative evaluators that measure how well machine-generated text can be distinguished from human-written text, as well as word overlap metrics that assess how similar the generated text compares to human-written references.
We determine to what extent these different evaluators agree on the ranking of a dozen of state-of-the-art generators for online product reviews.
We find that human evaluators do not correlate well with discriminative evaluators, leaving a bigger question of whether adversarial accuracy is the correct objective for natural language generation.

Built on

Unifying human and statistical evaluation for natural language generation
Original
Tatsunori B Hashimoto, Hugh Zhang, and Percy Liang. 2019 · 1904
Earlier work this paper cites.
Jointly measuring diversity and quality in text generation models
Original
Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. 2019 · 1904
Earlier work this paper cites.
The proof and measurement of association between two things
Charles Spearman. 1904 · 1904
Earlier work this paper cites.
Computing machinery and intelligence-am turing
Computing Machinery. 1950 · 1950
Earlier work this paper cites.
Tests for rank correlation coefficients. i
Edgar C Fieller, Herman O Hartley, and Egon S Pearson. 1957 · 1957
Earlier work this paper cites.
Perplexity?a measure of the difficulty of speech recognition tasks
Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977 · 1977
Earlier work this paper cites.
Applied nonparametric statistics
Wayne W Daniel et al. 1978 · 1978
Earlier work this paper cites.
A learning algorithm for continually running fully recurrent neural networks
Ronald J Williams and David Zipser. 1989 · 1989
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams. 1992 · 1992
Earlier work this paper cites.
Learning long-term dependencies with gradient descent is difficult
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994 · 1994
Earlier work this paper cites.
Support-vector networks
Corinna Cortes and Vladimir Vapnik. 1995 · 1995
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
Long short-term memory
Jürgen Schmidhuber and Sepp Hochreiter. 1997 · 1997
Earlier work this paper cites.
Towards evaluation in natural language generation
Robert Dale and Chris Mellish. 1998 · 1998
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998 · 1998
Earlier work this paper cites.
Evaluation in the context of natural language generation
Chris Mellish and Robert Dale. 1998 · 1998
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000 · 2000
Earlier work this paper cites.
Gradient flow in recurrent nets: the difficulty of learning long-term dependencies
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al. 2001 · 2001
Earlier work this paper cites.
Using a randomised controlled clinical trial to evaluate an nlg system
Ehud Reiter, Roma Robertson, A Scott Lennox, and Liesl Osman. 2001 · 2001
Earlier work this paper cites.
An empirical study of the naive bayes classifier
Irina Rish. 2001 · 2001
Earlier work this paper cites.
Classification and regression by randomforest
Andy Liaw, Matthew Wiener, et al. 2002 · 2002
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
Chin-Yew Lin. 2004 · 2004
Earlier work this paper cites.
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments
Satanjeev Banerjee and Alon Lavie. 2005 · 2005
Earlier work this paper cites.
Re-evaluation the role of bleu in machine translation research
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006 · 2006
Earlier work this paper cites.
Trec: An overview
Donna K Harman and Ellen M Voorhees. 2006 · 2006
Earlier work this paper cites.
Methodology of superiority vs. equivalence trials and non-inferiority trials
Erik Christensen. 2007 · 2007
Earlier work this paper cites.
Further meta-evaluation of machine translation
Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008 · 2008
Earlier work this paper cites.
Can we evaluate the quality of generated text?
David Hardcastle and Donia Scott. 2008 · 2008
Earlier work this paper cites.
The NIST 2008 metrics for machine translation challenge - overview, methodology, metrics, and results
Mark A. Przybocki, Kay Peterson, Sebastien Bronsart, and Gregory A. Sanders. 2009 · 2008
Earlier work this paper cites.
Curriculum learning
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009 · 2009
Earlier work this paper cites.
An investigation into the validity of some metrics for automatically evaluating natural language generation systems
Ehud Reiter and Anja Belz. 2009 · 2009
Earlier work this paper cites.
Why does unsupervised pre-training help deep learning?
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010 · 2010
Earlier work this paper cites.
Extensions of recurrent neural network language model
Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2011a · 2011
Earlier work this paper cites.

Similar

Rnnlm-recurrent neural network language modeling toolkit
Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Cernocky. 2011b · 2011
Cited alongside, same era.
Finding deceptive opinion spam by any stretch of the imagination
Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T Hancock. 2011 · 2011
Cited alongside, same era.
A survey of monte carlo tree search methods
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012 · 2012
Cited alongside, same era.
Putting human assessments of machine translation systems in order
Adam Lopez. 2012 · 2012
Cited alongside, same era.
Context dependent recurrent neural network language model
Tomas Mikolov and Geoffrey Zweig. 2012 · 2012
Cited alongside, same era.
Text-based measures of document diversity
Kevin Bache, David Newman, and Padhraic Smyth. 2013 · 2013
Cited alongside, same era.
One billion word benchmark for measuring progress in statistical language modeling
Original
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013 · 2013
Cited alongside, same era.
Statistical methods for rates and proportions
Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. 2013 · 2013
Cited alongside, same era.
Generating sequences with recurrent neural networks
Original
Alex Graves. 2013 · 2013
Cited alongside, same era.
Auto-encoding variational bayes
Original
Diederik P Kingma and Max Welling. 2013 · 2013
Cited alongside, same era.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Cited alongside, same era.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014 · 2014
Cited alongside, same era.

Then

Maximum-likelihood augmented discrete generative adversarial networks
Original
Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. 2017 · 2017
Later among the works it cites.
Towards diverse and natural image descriptions via a conditional gan
Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. 2017 · 2017
Later among the works it cites.
Learning to generate product reviews from attributes
Li Dong, Shaohan Huang, Furu Wei, Mirella Lapata, Ming Zhou, and Ke Xu. 2017 · 2017
Later among the works it cites.
Controlling linguistic style aspects in neural language generation
Jessica Ficler and Yoav Goldberg. 2017 · 2017
Later among the works it cites.
Adversarial evaluation of dialogue models
Original
Anjuli Kannan and Oriol Vinyals. 2017 · 2017
Later among the works it cites.
Adversarial learning for neural dialogue generation
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017 · 2017
Later among the works it cites.
Adversarial ranking for language generation
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. 2017 · 2017
Later among the works it cites.
Towards an automatic turing test: Learning to evaluate dialogue responses
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017 · 2017
Later among the works it cites.
Deep photo style transfer
Original
Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017 · 2017
Later among the works it cites.
Practical black-box attacks against machine learning
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. 2017 · 2017
Later among the works it cites.
Adversarial generation of natural language
Original
Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Pal, and Aaron Courville. 2017 · 2017
Later among the works it cites.
Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation
Original
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017 · 2017
Later among the works it cites.
Semi and weakly supervised semantic segmentation using generative adversarial network
Original
Nasim Souly, Concetto Spampinato, and Mubarak Shah. 2017 · 2017
Later among the works it cites.
Veegan: Reducing mode collapse in gans using implicit variational learning
Akash Srivastava, Lazar Valkoz, Chris Russell, Michael U Gutmann, and Charles Sutton. 2017 · 2017
Later among the works it cites.
Seqgan: Sequence generative adversarial nets with policy gradient
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017 · 2017
Later among the works it cites.
Adversarial feature matching for text generation
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017 · 2017
Later among the works it cites.
Unpaired image-to-image translation using cycle-consistent adversarial networks
Original
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017 · 2017
Later among the works it cites.
Language gans falling short
Original
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2018 · 2018
Later among the works it cites.
Eval all, trust a few, do wrong to none: Comparing sentence generation models
Original
Ondřej Cífka, Aliaksei Severyn, Enrique Alfonseca, and Katja Filippova. 2018 · 2018
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Later among the works it cites.
Maskgan: Better text generation via filling in the _
Original
William Fedus, Ian Goodfellow, and Andrew M Dai. 2018 · 2018
Later among the works it cites.
Language modeling for morphologically rich languages: Character-aware modeling for word-level prediction
Daniela Gerz, Ivan Vulić, Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018 · 2018
Later among the works it cites.
Long text generation via adversarial training with leaked information
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. 2018 · 2018
Later among the works it cites.
Generating sentences by editing prototypes
Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018 · 2018
Later among the works it cites.
Neural text generation: Past, present and beyond
Original
Sidi Lu, Yaoming Zhu, Weinan Zhang, Jun Wang, and Yong Yu. 2018 · 2018
Later among the works it cites.
Unsupervised learning of sentence embeddings using compositional n-gram features
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018 · 2018
Later among the works it cites.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018
Later among the works it cites.
On accurate evaluation of gans for language generation
Original
Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. 2018 · 2018
Later among the works it cites.
Toward diverse text generation with inverse reinforcement learning
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. 2018 · 2018
Later among the works it cites.
Evaluating text gans as language models
Original
Guy Tevet, Gavriel Habib, Vered Shwartz, and Jonathan Berant. 2018 · 2018
Later among the works it cites.
Hierarchical text generation and planning for strategic dialogue
Denis Yarats and Mike Lewis. 2018 · 2018
Later among the works it cites.
Texygen: A benchmarking platform for text generation models
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018 · 2018
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019
Closest in time.
Show, attend and tell: Neural image caption generation with visual attention
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015 · 2057
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…