2021

BARTScore: Evaluating Generated Text as Text Generation

Yuan, Weizhe, Neubig, Graham, Liu, Pengfei

Understand

A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation.

One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective.
In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models.
The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better.

Built on

A new measure of rank correlation
M. G. Kendall · 1938
Earlier work this paper cites.
A statistical interpretation of term specificity and its application in retrieval
Karen Sparck Jones · 1972
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002
Earlier work this paper cites.
Automatic evaluation of summaries using n-gram co-occurrence statistics
Chin-Yew Lin and Eduard Hovy · 2003
Earlier work this paper cites.
Statistical significance tests for machine translation evaluation
Philipp Koehn · 2004
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
Chin-Yew Lin · 2004
Earlier work this paper cites.
Evaluating content selection in summarization: The pyramid method
Ani Nenkova and Rebecca Passonneau · 2004
Earlier work this paper cites.
Overview of duc 2005
Hoa Trang Dang · 2005
Earlier work this paper cites.
Spearman rank correlation
Jerrold H Zar · 2005
Earlier work this paper cites.
Statistics (international student edition)
David Freedman, Robert Pisani, and Roger Purves · 2007
Earlier work this paper cites.
Supervised automatic evaluation for summarization with voted regression model
Tsutomu Hirao, Manabu Okumura, Norihito Yasuda, and Hideki Isozaki · 2007
Earlier work this paper cites.
Statistical machine translation
Philipp Koehn · 2009
Earlier work this paper cites.
Extending the METEOR machine translation evaluation metric to the phrase level
Michael Denkowski and Alon Lavie · 2010
Earlier work this paper cites.
Phrase-based statistical language generation using graphical models and active learning
François Mairesse, Milica Gasic, Filip Jurcicek, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young · 2010
Earlier work this paper cites.
BEER: BEtter evaluation as ranking
Miloš Stanojević and Khalil Sima’an · 2014
Earlier work this paper cites.
Semi-supervised sequence learning
Andrew M Dai and Quoc V Le · 2015
Earlier work this paper cites.
Teaching machines to read and comprehend
K. Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and P. Blunsom · 2015
Earlier work this paper cites.
chrF: character n-gram F-score for automatic MT evaluation
Maja Popović · 2015
Earlier work this paper cites.
Semantically conditioned lstm-based natural language generation for spoken dialogue systems
Original
Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young · 2015
Earlier work this paper cites.
Learning language representations for typology prediction
Chaitanya Malaviya, Graham Neubig, and Patrick Littell · 2017
Earlier work this paper cites.
Learning to score system summaries for better content selection evaluation
Maxime Peyrard, Teresa Botschen, and Iryna Gurevych · 2017
Earlier work this paper cites.
Get to the point: Summarization with pointer-generator networks
Abigail See, Peter J. Liu, and Christopher D. Manning · 2017
Earlier work this paper cites.
Fast abstractive summarization with reinforce-selected sentence rewriting
Yen-Chun Chen and Mohit Bansal · 2018
Earlier work this paper cites.
BanditSum: Extractive summarization as a contextual bandit
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung · 2018
Earlier work this paper cites.
Bottom-up abstractive summarization
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush · 2018
Earlier work this paper cites.
Bottom-up abstractive summarization
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush · 2018
Earlier work this paper cites.
Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies
Max Grusky, Mor Naaman, and Yoav Artzi · 2018
Earlier work this paper cites.

Similar

Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies
Max Grusky, Mor Naaman, and Yoav Artzi · 2018
Cited alongside, same era.
Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance
Qingsong Ma, Ondřej Bojar, and Yvette Graham · 2018
Cited alongside, same era.
Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization
Shashi Narayan, Shay B Cohen, and Mirella Lapata · 2018
Cited alongside, same era.
Ranking sentences for extractive summarization with reinforcement learning
Shashi Narayan, Shay B. Cohen, and Mirella Lapata · 2018
Cited alongside, same era.
GLUE: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman · 2018
Cited alongside, same era.
Neural document summarization by jointly learning to score and select sentences
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao · 2018
Cited alongside, same era.
Learning to retrieve reasoning paths over wikipedia graph for question answering
Original
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong · 2019
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Cited alongside, same era.
Unified language model pre-training for natural language understanding and generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon · 2019
Cited alongside, same era.
Ranking generated summaries by correctness: An interesting but challenging application for natural language inference
Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych · 2019
Cited alongside, same era.
Large-scale, diverse, paraphrastic bitexts via sampling and clustering
J. Edward Hu, Abhinav Singh, Nils Holzenberger, Matt Post, and Benjamin Van Durme · 2019
Cited alongside, same era.
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer · 2019
Cited alongside, same era.

Then

A joint neural model for information extraction with global features
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu · 2020
Later among the works it cites.
Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics
Nitika Mathur, Timothy Baldwin, and Trevor Cohn · 2020
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2020
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2020
Later among the works it cites.
COMET: A neural framework for MT evaluation
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie · 2020
Later among the works it cites.
Exploiting cloze questions for few-shot text classification and natural language inference
Original
Timo Schick and Hinrich Schütze · 2020
Later among the works it cites.
Few-shot text generation with pattern-exploiting training
Original
Timo Schick and Hinrich Schütze · 2020
Later among the works it cites.
It’s not just size that matters: Small language models are also few-shot learners
Original
Timo Schick and Hinrich Schütze · 2020
Later among the works it cites.
BLEURT: Learning robust metrics for text generation
Thibault Sellam, Dipanjan Das, and Ankur Parikh · 2020
Later among the works it cites.
Autoprompt: Eliciting knowledge from language models with automatically generated prompts
Original
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh · 2020
Later among the works it cites.
Automatic machine translation evaluation in many languages via zero-shot paraphrasing
Brian Thompson and Matt Post · 2020
Later among the works it cites.
Asking and answering questions to evaluate the factual consistency of summaries
Alex Wang, Kyunghyun Cho, and M. Lewis · 2020
Later among the works it cites.
Heterogeneous graph neural networks for extractive document summarization
Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang · 2020
Later among the works it cites.
Transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush · 2020
Later among the works it cites.
Pegasus: Pre-training with extracted gap-sentences for abstractive summarization
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu · 2020
Later among the works it cites.
Bertscore: Evaluating text generation with bert
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi · 2020
Later among the works it cites.
Extractive summarization as text matching
Original
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang · 2020
Later among the works it cites.
Extractive summarization as text matching
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang · 2020
Later among the works it cites.
GSum: A general framework for guided neural abstractive summarization
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig · 2021
Closest in time.
Summeval: Re-evaluating summarization evaluation
A. R. Fabbri, Wojciech Kryscinski, Bryan McCann, R. Socher, and Dragomir Radev · 2021
Closest in time.
How many data points is a prompt worth?
Teven Le Scao and Alexander Rush · 2021
Closest in time.
Prefix-tuning: Optimizing continuous prompts for generation
Original
Xiang Lisa Li and Percy Liang · 2021
Closest in time.
Explainaboard: An explainable leaderboard for nlp
Original
Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaicheng Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, and Graham Neubig · 2021
Closest in time.
Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, 2021
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig · 2021
Closest in time.
Simcls: A simple framework for contrastive learning of abstractive summarization
Original
Yixin Liu and Pengfei Liu · 2021
Closest in time.
Prompt programming for large language models: Beyond the few-shot paradigm
Laria Reynolds and Kyle McDonell · 2021
Closest in time.
Can we automate scientific reviewing?
Original
Weizhe Yuan, Pengfei Liu, and Graham Neubig · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…