CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Bertscore: Evaluating text generation with bert

Original

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019 · 1904

Earlier work this paper cites.

Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons

Original

Margaret Li, Jason Weston, and Stephen Roller. 2019 · 1909

Earlier work this paper cites.

Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance

Original

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019 · 1909

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

Evaluating content selection in summarization: The pyramid method

Ani Nenkova and Rebecca J Passonneau. 2004 · 2004

Earlier work this paper cites.

Inter-annotator agreement

Ron Artstein. 2017 · 2017

Earlier work this paper cites.

Why we need new evaluation metrics for NLG

Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017 · 2017

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Earlier work this paper cites.

Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations

Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019 · 2019

Earlier work this paper cites.

A topic augmented text generation model: Joint learning of semantics and structural features

Hongyin Tang, Miao Li, and Beihong Jin. 2019 · 2019

Earlier work this paper cites.

Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing

Anya Belz, Simon Mille, and David M. Howcroft. 2020 · 2020

Earlier work this paper cites.

Beyond accuracy: Behavioral testing of NLP models with CheckList

Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020 · 2020

Earlier work this paper cites.

Summeval: Re-evaluating summarization evaluation

Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021 · 2021

Earlier work this paper cites.

Perception score: A learned metric for open-ended text generation evaluation

Jing Gu, Qingyang Wu, and Zhou Yu. 2021 · 2021

Earlier work this paper cites.

Bartscore: Evaluating generated text as text generation

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021 · 2021

Earlier work this paper cites.

Generating literal and implied subquestions to fact-check complex claims

Original

Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Cited alongside, same era.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 · 2022

Cited alongside, same era.

Generating scientific claims for zero-shot scientific fact checking

Original

Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. 2022 · 2022

Cited alongside, same era.

Towards a unified multi-dimensional evaluator for text generation

Original

Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022 · 2022

Cited alongside, same era.

Gpt-4 technical report

Original

Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks

Original

Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K. Surikuchi, Ece Takmaz, and Alberto Testoni. 2024 · 2024

Closest in time.

Chateval: Towards better LLM-based evaluators through multi-agent debate

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024 · 2024

Closest in time.

The llama 3 herd of models

Original

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024 · 2024

Closest in time.

Analyzing and evaluating correlation measures in nlg meta-evaluation

Original

Mingqi Gao, Xinyu Hu, Li Lin, and Xiaojun Wan. 2024 · 2024

Closest in time.