Human-like Summarization Evaluation with ChatGPT

Bertscore: Evaluating text generation with bert

Original

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020 · 1904

Earlier work this paper cites.

ROUGE: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

Evaluating content selection in summarization: The pyramid method

Ani Nenkova and Rebecca Passonneau. 2004 · 2004

Earlier work this paper cites.

Language models are few-shot learners

Original

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2005

Earlier work this paper cites.

Learning to summarize from human feedback

Original

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022 · 2009

Earlier work this paper cites.

Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies

Max Grusky, Mor Naaman, and Yoav Artzi. 2018 · 2018

Earlier work this paper cites.

MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance

Original

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019 · 2019

Earlier work this paper cites.

Re-evaluating evaluation in text summarization

Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020 · 2020

Cited alongside, same era.

FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization

Esin Durmus, He He, and Mona Diab. 2020 · 2020

Cited alongside, same era.

Evaluating the factual consistency of abstractive text summarization

Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020 · 2020

Cited alongside, same era.

Asking and answering questions to evaluate the factual consistency of summaries

Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020 · 2020

Cited alongside, same era.

The ReproGen shared task on reproducibility of human evaluations in NLG: Overview and results

Anya Belz, Anastasia Shimorina, Shubham Agarwal, and Ehud Reiter. 2021 · 2021

Cited alongside, same era.

Training language models to follow instructions with human feedback

Original

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022 · 2022

Later among the works it cites.

Exploring chatgpt’s ability to rank content: A preliminary study on consistency with human preferences

Original

Yunjie Ji, Yan Gong, Yiping Peng, Chao Ni, Peiyan Sun, Dongyu Pan, Baochang Ma, and Xiangang Li. 2023 · 2023

Closest in time.

Large language models are state-of-the-art evaluators of translation quality

Original

Tom Kocmi and Christian Federmann. 2023 · 2023

Closest in time.

Gpteval: Nlg evaluation using gpt-4 with better human alignment

Original

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 · 2023

Closest in time.