2020

A Survey of Evaluation Metrics Used for NLG Systems

Sai, Ananya B., Mohankumar, Akash Kumar, Khapra, Mitesh M.

Understand

The success of Deep Learning has created a surge in interest in a wide a range of Natural Language Generation (NLG) tasks.

Deep Learning has not only pushed the state of the art in several existing NLG tasks but has also facilitated researchers to explore various newer NLG tasks such as image captioning.
Such rapid progress in NLG has necessitated the development of accurate automatic evaluation metrics that would allow us to track the progress in the field of NLG.
However, unlike classification tasks, automatically evaluating NLG systems in itself is a huge challenge.

Built on

TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Original
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019 · 1901
Earlier work this paper cites.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
Original
Margaret Li, Jason Weston, and Stephen Roller. 2019 · 1909
Earlier work this paper cites.
Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation
Original
Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P. Parikh. 2019 · 1910
Earlier work this paper cites.
A technique for the measurement of attitudes
Rensis Likert. 1932 · 1932
Earlier work this paper cites.
I.—COMPUTING MACHINERY AND INTELLIGENCE
A. M. TURING. 1950 · 1950
Earlier work this paper cites.
From first conception to first demonstration: the nascent years of machine translation, 1947-1954 - A chronology
John Hutchins. 1997 · 1954
Earlier work this paper cites.
Regression Analysis
E.J. Williams. 1959 · 1959
Earlier work this paper cites.
ELIZA—a computer program for the study of natural language communication between man and machine
Joseph Weizenbaum. 1966 · 1966
Earlier work this paper cites.
Procedures as a representation for data in a computer program for understanding natural language
Terry Winograd. 1971 · 1971
Earlier work this paper cites.
Understanding natural language
Terry Winograd. 1972 · 1972
Earlier work this paper cites.
Indexing by Latent Semantic Analysis
Scott C. Deerwester, Susan T. Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. 1990 · 1990
Earlier work this paper cites.
Hierarchical Recurrent Neural Networks for Long-Term Dependencies. In Advances in Neural Information Processing Systems 8, NIPS, Denver, CO, USA, November 27-30, 1995 , David S. Touretzky, Michael Mozer, and Michael E. Hasselmo (Eds.). MIT Press, 493–499
Salah El Hihi and Yoshua Bengio. 1995 · 1995
Earlier work this paper cites.
A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge
Thomas K Landauer and Susan T Dumais. 1997 · 1997
Earlier work this paper cites.
Accelerated DP based search for statistical translation. In Fifth European Conference on Speech Communication and Technology, EUROSPEECH 1997, Rhodes, Greece, September 22-25, 1997 , George Kokkinakis, Nikos Fakotakis, and Evangelos Dermatas (Eds.). ISCA
Christoph Tillmann, Stephan Vogel, Hermann Ney, A. Zubiaga, and Hassan Sawaf. 1997 · 1997
Earlier work this paper cites.
Improving Summaries by Revising Them. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, College Park, Maryland, USA, 558–565
Inderjeet Mani, Barbara Gates, and Eric Bloedorn. 1999 · 1999
Earlier work this paper cites.
Evaluation Metrics for Generation. In INLG 2000 - Proceedings of the First International Natural Language Generation Conference, June 12-16, 2000, Mitzpe Ramon, Israel , Michael Elhadad (Ed.). The Association for Computer Linguistics, 1–8
Srinivas Bangalore, Owen Rambow, and Steve Whittaker. 2000 · 2000
Earlier work this paper cites.
Contextual word similarity
Ido Dagan. 2000 · 2000
Earlier work this paper cites.
An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research. In Proceedings of the Second International Conference on Language Resources and Evaluation, LREC 2000, 31 May - June 2, 2000, Athens, Greece . European Language Resources Association
Sonja Nießen, Franz Josef Och, Gregor Leusch, and Hermann Ney. 2000 · 2000
Earlier work this paper cites.
Summarization Evaluation: An Overview. In Proceedings of the Third Second Workshop Meeting on Evaluation of Chinese & Japanese Text Retrieval and Text Summarization, NTCIR-2, Tokyo, Japan, March 7-9, 2001 . National Institute of Informatics (NII)
Inderjeet Mani. 2001 · 2001
Earlier work this paper cites.
Automatic Evaluation of Machine Translation Quality Using N-Gram Co-Occurrence Statistics. In Proceedings of the Second International Conference on Human Language Technology Research (San Diego, California) (HLT ’02) . Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 138–145
George Doddington. 2002 · 2002
Earlier work this paper cites.
BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 311–318
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Correlating automated and human assessments of machine translation quality. In Proceedings of MT summit IX . Citeseer, 63–70
Deborah Coughlin. 2003 · 2003
Earlier work this paper cites.
An Experiment in Comparative Evaluation: Humans Vs. Computers
Andrei Popescu-Belis. 2003 · 2003
Earlier work this paper cites.
Evaluation of Machine Translation and its Evaluation
Joseph Turian, Luke Shen, and I. Melamed. 2003 · 2003
Earlier work this paper cites.
NUBIA: NeUral Based Interchangeability Assessor for Text Generation
Original
Hassan Kané, Muhammed Yusuf Kocyigit, Ali Abdalla, Pelkins Ajanoh, and Mohamed Coulibali. 2020 · 2004
Earlier work this paper cites.
ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In COLING 2004, 20th International Conference on Computational Linguistics, Proceedings of the Conference, 23-27 August 2004, Geneva, Switzerland
Chin-Yew Lin and Franz Josef Och. 2004 · 2004
Earlier work this paper cites.
ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out . Association for Computational Linguistics, Barcelona, Spain, 74–81
Chin-Yew Lin. 2004 · 2004
Earlier work this paper cites.
The power of outliers (and why researchers should always check for them)
Jason W Osborne and Amy Overbay. 2004 · 2004
Earlier work this paper cites.
WordNet: : Similarity - Measuring the Relatedness of Concepts. In Proceedings of the Nineteenth National Conference on Artificial Intelligence, Sixteenth Conference on Innovative Applications of Artificial Intelligence, July 25-29, 2004, San Jose, California, USA , Deborah L. McGuinness and George Ferguson (Eds.). AAAI Press / The MIT Press, 1024–1025
Ted Pedersen, Siddharth Patwardhan, and Jason Michelizzi. 2004 · 2004
Earlier work this paper cites.
BLEURT: Learning Robust Metrics for Text Generation
Original
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020 · 2004
Earlier work this paper cites.
Interpreting BLEU/NIST Scores: How Much Improvement do We Need to Have a Better System?. In Proceedings of the Fourth International Conference on Language Resources and Evaluation, LREC 2004, May 26-28, 2004, Lisbon, Portugal . European Language Resources Association
Ying Zhang, Stephan Vogel, and Alex Waibel. 2004 · 2004
Earlier work this paper cites.
Designing Precise and Robust Dialogue Response Evaluators
Original
Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020 · 2004
Earlier work this paper cites.
METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization . Association for Computational Linguistics, Ann Arbor, Michigan, 65–72
Satanjeev Banerjee and Alon Lavie. 2005 · 2005
Earlier work this paper cites.
Language Models are Few-Shot Learners
Original
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2005
Earlier work this paper cites.
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation
Original
Weixin Liang, James Zou, and Zhou Yu. 2020 · 2005
Earlier work this paper cites.
Learning an Unreferenced Metric for Online Dialogue Evaluation
Original
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L. Hamilton, and Joelle Pineau. 2020 · 2005
Earlier work this paper cites.
Evaluating Evaluation Methods for Generation in the Presence of Variation. In Proceedings of the 6th International Conference on Computational Linguistics and Intelligent Text Processing (Mexico City, Mexico) (CICLing’05) . Springer-Verlag, Berlin, Heidelberg, 341–351
Amanda Stent, Matthew Marge, and Mohit Singhai. 2005 · 2005
Earlier work this paper cites.
Some Issues in Automatic Evaluation of English-Hindi MT : More Blues for BLEU
Ananthakrishnan, Pushpak Bhattacharyya, Murugesan Sasikumar, and Ritesh M. Shah. 2006 · 2006
Earlier work this paper cites.
Speaker Sensitive Response Evaluation Model
Original
JinYeong Bak and Alice Oh. 2020 · 2006
Earlier work this paper cites.
Comparing Automatic and Human Evaluation of NLG Systems. In 11th Conference of the European Chapter of the Association for Computational Linguistics . Association for Computational Linguistics, Trento, Italy
Anja Belz and Ehud Reiter. 2006 · 2006
Earlier work this paper cites.
Re-evaluating the Role of Bleu in Machine Translation Research. In 11th Conference of the European Chapter of the Association for Computational Linguistics . Association for Computational Linguistics, Trento, Italy
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006 · 2006
Earlier work this paper cites.
CDER: Efficient MT Evaluation Using Block Movements. In EACL 2006, 11st Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, April 3-7, 2006, Trento, Italy , Diana McCarthy and Shuly Wintner (Eds.). The Association for Computer Linguistics
Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006 · 2006
Earlier work this paper cites.
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics
Original
Nitika Mathur, Tim Baldwin, and Trevor Cohn. 2020 · 2006
Earlier work this paper cites.
A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas . 223–231
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006 · 2006
Earlier work this paper cites.
Factorization of Synchronous Context-Free Grammars in Linear Time. In Proceedings of the NAACL-HLT 2007 / AMTA Workshop on Syntax and Structure in Statistical Translation, SSST@HLT-NAACL 2007, Rochester, New York, USA, April 26, 2007 , Dekai Wu and David Chiang (Eds.). Association for Computational Linguistics, 25–32
Hao Zhang and Daniel Gildea. 2007 · 2007
Earlier work this paper cites.
ROUGE-C: A Fully Automated Evaluation Method for Multi-document Summarization. In The 2008 IEEE International Conference on Granular Computing, GrC 2008, Hangzhou, China, 26-28 August 2008 . IEEE, 269–274
Tingting He, Jinguang Chen, Liang Ma, Zhuoming Gui, Fang Li, Wei Shao, and Qian Wang. 2008 · 2008
Earlier work this paper cites.
Definitional and human constraints on structural annotation of English
Geoffrey Sampson and Anna Babarczy. 2008 · 2008
Earlier work this paper cites.
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 , Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 563–578
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019 · 2008
Earlier work this paper cites.
Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL . ACL, 286–295
Chris Callison-Burch. 2009 · 2009
Earlier work this paper cites.
Introduction to algorithms
Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. 2009 · 2009
Earlier work this paper cites.
An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems
Ehud Reiter and Anja Belz. 2009 · 2009
Earlier work this paper cites.
Evaluation Measures for Text Summarization
Josef Steinberger and Karel Jezek. 2009 · 2009
Earlier work this paper cites.
METEOR-NEXT and the METEOR Paraphrase Tables: Improved Evaluation Support for Five Target Languages. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, WMT@ACL 2010, Uppsala, Sweden, July 15-16, 2010 , Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, and Omar Zaidan (Eds.). Association for Computational Linguistics, 339–342
Michael J. Denkowski and Alon Lavie. 2010 · 2010
Earlier work this paper cites.
Introducing Shared Tasks to NLG: The TUNA Shared Task Evaluation Challenges. In Empirical Methods in Natural Language Generation: Data-oriented Methods and Empirical Evaluation (Lecture Notes in Computer Science, Vol. 5790) , Emiel Krahmer and Mariët Theune (Eds.). Springer, 264–293
Albert Gatt and Anja Belz. 2010 · 2010
Earlier work this paper cites.
Asiya: An Open Toolkit for Automatic Machine Translation (Meta-)Evaluation
Jesús Giménez and Lluís Màrquez. 2010 · 2010
Earlier work this paper cites.
Phrase-Based Statistical Language Generation Using Graphical Models and Active Learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Uppsala, Sweden, 1552–1561
François Mairesse, Milica Gašić, Filip Jurčíček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010 · 2010
Earlier work this paper cites.
Learning From Crowds
Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010 · 2010
Earlier work this paper cites.
Unsupervised Modeling of Twitter Conversations. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 2-4, 2010, Los Angeles, California, USA . The Association for Computational Linguistics, 172–180
Alan Ritter, Colin Cherry, and Bill Dolan. 2010 · 2010
Earlier work this paper cites.
MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA , Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (Eds.). The Association for Computer Linguistics, 220–229
Chi-kiu Lo and Dekai Wu. 2011 · 2011
Earlier work this paper cites.
Fully Automatic Semantic MT Evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, WMT@NAACL-HLT 2012, June 7-8, 2012, Montréal, Canada , Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia (Eds.). The Association for Computer Linguistics, 243–252
Chi-kiu Lo, Anand Karthik Tumuluru, and Dekai Wu. 2012 · 2012
Earlier work this paper cites.
Unsupervised vs. supervised weight estimation for semantic MT evaluation metrics. In Proceedings of the Sixth Workshop on Syntax, Semantics and Structure in Statistical Translation, SSST@ACT 2012, Jeju, Korea, 12 July 2012 , Marine Carpuat, Lucia Specia, and Dekai Wu (Eds.). Association for Computational Linguistics, 49–56
Chi-kiu Lo and Dekai Wu. 2012 · 2012
Earlier work this paper cites.

Similar

A Comparison of Greedy and Optimal Assessment of Natural Language Student Input Using Word-to-Word Similarity Metrics. In BEA@NAACL-HLT
Vasile Rus and Mihai C. Lintean. 2012 · 2012
Cited alongside, same era.
Image Description using Visual Dependency Representations. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL . ACL, 1292–1302
Desmond Elliott and Frank Keller. 2013 · 2013
Cited alongside, same era.
Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, LAW-ID@ACL 2013, August 8-9, 2013, Sofia, Bulgaria , Stefanie Dipper, Maria Liakata, and Antonio Pareja-Lora (Eds.). The Association for Computer Linguistics, 33–41
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013 · 2013
Cited alongside, same era.
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013 · 2013
Cited alongside, same era.
Results of the WMT13 Metrics Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, WMT@ACL 2013, August 8-9, 2013, Sofia, Bulgaria . The Association for Computer Linguistics
Matous Machácek and Ondrej Bojar. 2013 · 2013
Cited alongside, same era.
Efficient Estimation of Word Representations in Vector Space
Original
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 · 2013
Cited alongside, same era.
Findings of the 2014 Workshop on Statistical Machine Translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation . Association for Computational Linguistics, Baltimore, Maryland, USA, 12–58
Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014 · 2014
Cited alongside, same era.
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Original
Junyoung Chung, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Cited alongside, same era.
Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26-27, 2014, Baltimore, Maryland, USA . The Association for Computer Linguistics, 376–380
Michael J. Denkowski and Alon Lavie. 2014 · 2014
Cited alongside, same era.
Bootstrapping Dialog Systems with Word Embeddings
Gabriel Forgues and Joelle Pineau. 2014 · 2014
Cited alongside, same era.
XMEANT: Better semantic MT evaluation without reference translations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers . The Association for Computer Linguistics, 765–771
Chi-kiu Lo, Meriem Beloucif, Markus Saers, and Dekai Wu. 2014 · 2014
Cited alongside, same era.
Results of the WMT14 Metrics Shared Task. In Proceedings of the Ninth Workshop on Statistical Machine Translation . Association for Computational Linguistics, Baltimore, Maryland, USA, 293–301
Matouš Macháček and Ondřej Bojar. 2014 · 2014
Cited alongside, same era.

Then

Universal Sentence Encoder
Original
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018 · 2018
Later among the works it cites.
The price of debiasing automatic metrics in natural language evaluation
Original
Arun Tejasvi Chaganty, Stephen Mussmann, and Percy Liang. 2018 · 2018
Later among the works it cites.
Learning to Evaluate Image Captioning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 . IEEE Computer Society, 5804–5812
Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge J. Belongie. 2018 · 2018
Later among the works it cites.
Deep learning in natural language processing
Li Deng and Yang Liu. 2018 · 2018
Later among the works it cites.
Understanding Back-Translation at Scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018 , Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, 489–500
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018 · 2018
Later among the works it cites.
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation
Albert Gatt and Emiel Krahmer. 2018 · 2018
Later among the works it cites.
Meteor++: Incorporating Copy Knowledge into Machine Translation Evaluation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers . Association for Computational Linguistics, Belgium, Brussels, 740–745
Yinuo Guo, Chong Ruan, and Junfeng Hu. 2018 · 2018
Later among the works it cites.
The NarrativeQA Reading Comprehension Challenge
Tomás Kociský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018 · 2018
Later among the works it cites.
The challenging task of summary evaluation: an overview
Elena Lloret, Laura Plaza, and Ahmet Aker. 2018 · 2018
Later among the works it cites.
An efficient framework for learning sentence representations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net
Lajanugen Logeswaran and Honglak Lee. 2018 · 2018
Later among the works it cites.
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers . Association for Computational Linguistics, Belgium, Brussels, 671–688
Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018 · 2018
Later among the works it cites.
Towards a Better Metric for Evaluating Question Generation Systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018 , Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, 3950–3959
Preksha Nema and Mitesh M. Khapra. 2018 · 2018
Later among the works it cites.
SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge. In Proceedings of The 12th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018 , Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). Association for Computational Linguistics, 747–757
Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018 · 2018
Later among the works it cites.
ITER: Improving Translation Edit Rate through Optimizable Edit Costs. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, WMT 2018, Belgium, Brussels, October 31 - November 1, 2018 , Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana L. Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (Eds.). Association for Computational Linguistics, 746–750
Joybrata Panja and Sudip Kumar Naskar. 2018 · 2018
Later among the works it cites.
Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers) , Marilyn A. Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, 2227–2237
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 · 2018
Later among the works it cites.
A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 - November 1, 2018 , Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana L. Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (Eds.). Association for Computational Linguistics, 186–191
Matt Post. 2018 · 2018
Later among the works it cites.
A Structured Review of the Validity of BLEU
Ehud Reiter. 2018 · 2018
Later among the works it cites.
Learning-based Composite Metrics for Improved Caption Evaluation. In Proceedings of ACL 2018, Melbourne, Australia, July 15-20, 2018, Student Research Workshop , Vered Shwartz, Jeniya Tabassum, Rob Voigt, Wanxiang Che, Marie-Catherine de Marneffe, and Malvina Nissim (Eds.). Association for Computational Linguistics, 14–20
Naeha Sharif, Lyndon White, Mohammed Bennamoun, and Syed Afaq Ali Shah. 2018a · 2018
Later among the works it cites.
NNEval: Neural Network Based Evaluation Metric for Image Captioning. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII (Lecture Notes in Computer Science, Vol. 11212) , Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer, 39–55
Naeha Sharif, Lyndon White, Mohammed Bennamoun, and Syed Afaq Ali Shah. 2018b · 2018
Later among the works it cites.
RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, WMT 2018, Belgium, Brussels, October 31 - November 1, 2018 , Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana L. Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (Eds.). Association for Computational Linguistics, 751–758
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018 · 2018
Later among the works it cites.
RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , Sheila A. McIlraith and Kilian Q. Weinberger (Eds.). AAAI Press, 722–729
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018 · 2018
Later among the works it cites.
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, Melbourne, Australia, 451–462
John Wieting and Kevin Gimpel. 2018 · 2018
Later among the works it cites.
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) . Association for Computational Linguistics, New Orleans, Louisiana, 1112–1122
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018 · 2018
Later among the works it cites.
Recent trends in deep learning based natural language processing
Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018 · 2018
Later among the works it cites.
Personalizing Dialogue Agents: I have a dog, do you have pets too?. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, Melbourne, Australia, 2204–2213
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018 · 2018
Later among the works it cites.
Evaluating Question Answering Evaluation. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, MRQA@EMNLP 2019, Hong Kong, China, November 4, 2019 , Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen (Eds.). Association for Computational Linguistics, 119–124
Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019 · 2019
Later among the works it cites.
WMDO: Fluency-based Word Mover’s Distance for Machine Translation Evaluation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) . Association for Computational Linguistics, Florence, Italy, 494–500
Julian Chow, Lucia Specia, and Pranava Madhyastha. 2019 · 2019
Later among the works it cites.
Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , Anna Korhonen, David R. Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, 2748–2760
Elizabeth Clark, Asli Çelikyilmaz, and Noah A. Smith. 2019 · 2019
Later among the works it cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171–4186
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Later among the works it cites.
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , Anna Korhonen, David R. Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, 4884–4895
Bhuwan Dhingra, Manaal Faruqui, Ankur P. Parikh, Ming-Wei Chang, Dipanjan Das, and William W. Cohen. 2019 · 2019
Later among the works it cites.
Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge
Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. 2020 · 2019
Later among the works it cites.
Word Embedding-Based Automatic MT Evaluation Metric using Word Position Information. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Association for Computational Linguistics, Minneapolis, Minnesota, 1874–1883
Hiroshi Echizen’ya, Kenji Araki, and Eduard Hovy. 2019 · 2019
Later among the works it cites.
Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments
Marina Fomicheva and Lucia Specia. 2019 · 2019
Later among the works it cites.
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 13658–13669
Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Àgata Lapedriza, and Rosalind W. Picard. 2019 · 2019
Later among the works it cites.
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings
Original
Sarik Ghazarian, Johnny Tian-Zheng Wei, Aram Galstyan, and Nanyun Peng. 2019 · 2019
Later among the works it cites.
Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation. In Proceedings of the Fourth Conference on Machine Translation, WMT 2019, Florence, Italy, August 1-2, 2019 - Volume 2: Shared Task Papers, Day 1 , Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana L. Neves, Matt Post, Marco Turchi, and Karin Verspoor (Eds.). Association for Computational Linguistics, 501–506
Yinuo Guo and Junfeng Hu. 2019 · 2019
Later among the works it cites.
Evaluating Rewards for Question Generation Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 2278–2283
Tom Hosking and Sebastian Riedel. 2019 · 2019
Later among the works it cites.
Neural Text Summarization: A Critical Evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Association for Computational Linguistics, Hong Kong, China, 540–551
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019 · 2019
Later among the works it cites.
Reasoning Over Paragraph Effects in Situations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, MRQA@EMNLP 2019, Hong Kong, China, November 4, 2019 , Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen (Eds.). Association for Computational Linguistics, 58–62
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019 · 2019
Later among the works it cites.
YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources. In Proceedings of the Fourth Conference on Machine Translation, WMT 2019, Florence, Italy, August 1-2, 2019 - Volume 2: Shared Task Papers, Day 1 , Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana L. Neves, Matt Post, Marco Turchi, and Karin Verspoor (Eds.). Association for Computational Linguistics, 507–513
Chi-kiu Lo. 2019 · 2019
Later among the works it cites.
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges. In Proceedings of the Fourth Conference on Machine Translation, WMT 2019, Florence, Italy, August 1-2, 2019 - Volume 2: Shared Task Papers, Day 1 , Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana L. Neves, Matt Post, Marco Turchi, and Karin Verspoor (Eds.). Association for Computational Linguistics, 62–90
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019 · 2019
Later among the works it cites.
Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , Anna Korhonen, David R. Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, 2799–2808
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019 · 2019
Later among the works it cites.
Re-Evaluating ADEM: A Deeper Look at Scoring Dialogue Responses. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019 . AAAI Press, 6220–6227
Ananya B. Sai, Mithun Das Gupta, Mitesh M. Khapra, and Mukundhan Srinivasan. 2019 · 2019
Later among the works it cites.
What makes a good conversation? How controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 1702–1723
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019 · 2019
Later among the works it cites.
Machine Translation Evaluation with BERT Regressor
Original
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2019 · 2019
Later among the works it cites.
Webnlg challenge: Human evaluation results
Anastasia Shimorina, Claire Gardent, Shashi Narayan, and Laura Perez-Beltrachini. 2019 · 2019
Later among the works it cites.
EED: Extended Edit Distance Measure for Machine Translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) . Association for Computational Linguistics, Florence, Italy, 514–520
Peter Stanchev, Weiyue Wang, and Hermann Ney. 2019 · 2019
Later among the works it cites.
Beyond BLEU: Training Neural Machine Translation with Semantic Similarity. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , Anna Korhonen, David R. Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, 4344–4355
John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019 · 2019
Later among the works it cites.
XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 5754–5764
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019 · 2019
Later among the works it cites.
BERTScore: Evaluating Text Generation with BERT
Original
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019 · 2019
Later among the works it cites.
Linguistic Appropriateness and Pedagogic Usefulness of Reading Comprehension Questions. In Proceedings of The 12th Language Resources and Evaluation Conference . European Language Resources Association, Marseille, France, 1753–1762
Andrea Horbach, Itziar Aldabe, Marie Bexte, Oier Lopez de Lacalle, and Montse Maritxalar. 2020 · 2020
Closest in time.
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR
Original
Juri Opitz and Anette Frank. 2020 · 2020
Closest in time.
How To Evaluate Your Dialogue System: Probe Tasks as an Alternative for Token-level Evaluation Metrics
Original
Prasanna Parthasarathi, Joelle Pineau, and Sarath Chandar. 2020 · 2020
Closest in time.
Query-based summarization of discussion threads
Suzan Verberne, Emiel Krahmer, Sander Wubben, and Antal van den Bosch. 2020 · 2020
Closest in time.
Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA - Short Papers . The Association for Computer Linguistics, 176–181
Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011 · 2031
Closest in time.
Adversarial Examples for Evaluating Reading Comprehension Systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017 , Martha Palmer, Rebecca Hwa, and Sebastian Riedel (Eds.). Association for Computational Linguistics, 2021–2031
Robin Jia and Percy Liang. 2017 · 2031
Closest in time.
Discrete vs. Continuous Rating Scales for Language Evaluation in NLP. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA - Short Papers . The Association for Computer Linguistics, 230–235
Anja Belz and Eric Kow. 2011 · 2040
Closest in time.
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 (JMLR Workshop and Conference Proceedings, Vol. 37) , Francis R. Bach and David M. Blei (Eds.). JMLR.org, 2048–2057
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015 · 2057
Closest in time.
A New Quantitative Quality Measure for Machine Translation Systems. In COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
Keh-Yih Su, Ming-Wen Wu, and Jing-Shin Chang. 1992 · 2067
Closest in time.
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 2: Short Papers . The Association for Computer Linguistics, 445–450
Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015 · 2073
Closest in time.
Comparing Automatic Evaluation Measures for Image Description. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers . The Association for Computer Linguistics, 452–457
Desmond Elliott and Frank Keller. 2014 · 2074
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…