LLM-based NLG Evaluation: Current Status and Challenges

Chatgpt to replace crowdsourcing of paraphrases for intent classification: Higher diversity and comparable model robustness

Cegin, Ján, Jakub Simko, and Peter Brusilovsky. 2023 · 1905

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation

Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

Interpreting BLEU/NIST scores: How much improvement do we need to have a better system?

Zhang, Ying, Stephan Vogel, and Alex Waibel. 2004 · 2004

Earlier work this paper cites.

Evaluating evaluation methods for generation in the presence of variation

Stent, Amanda, Matthew Marge, and Mohit Singhai. 2005 · 2005

Earlier work this paper cites.

Evaluation of text generation: A survey

Original

Celikyilmaz, Asli, Elizabeth Clark, and Jianfeng Gao. 2020 · 2006

Earlier work this paper cites.

Challenges in data-to-document generation

Wiseman, Sam, Stuart M. Shieber, and Alexander M. Rush. 2017 · 2017

Earlier work this paper cites.

Language (technology) is power: A critical survey of "bias" in NLP

Blodgett, Su Lin, Solon Barocas, Hal Daumé III, and Hanna M. Wallach. 2020 · 2020

Earlier work this paper cites.

FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization

Durmus, Esin, He He, and Mona T. Diab. 2020 · 2020

Earlier work this paper cites.

Evaluating large language models trained on code

Chen, Mark, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021 · 2021

Earlier work this paper cites.

Survey on evaluation methods for dialogue systems

Deriu, Jan, Álvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021 · 2021

Earlier work this paper cites.

Summeval: Re-evaluating summarization evaluation

Fabbri, Alexander R., Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2021 · 2021

Earlier work this paper cites.

The GEM benchmark: Natural language generation, its evaluation and metrics

Gehrmann, Sebastian, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey Osei, Ankur P. Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021 · 2021

Earlier work this paper cites.

Human evaluation of creative NLG systems: An interdisciplinary survey on recent papers

Original

Hämäläinen, Mika and Khalid Al-Najjar. 2021 · 2021

Earlier work this paper cites.

Human evaluation of automatically generated text: Current trends and best practice guidelines

van der Lee, Chris, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021 · 2021

Earlier work this paper cites.

Webgpt: Browser-assisted question-answering with human feedback

Original

Nakano, Reiichiro, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Perturbation checklists for evaluating NLG evaluation metrics

Sai, Ananya B., Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. 2021 · 2021

Earlier work this paper cites.

Bartscore: Evaluating generated text as text generation

Yuan, Weizhe, Graham Neubig, and Pengfei Liu. 2021 · 2021

Earlier work this paper cites.

A human-machine collaborative framework for evaluating malevolence in dialogues

Zhang, Yangjun, Pengjie Ren, and Maarten de Rijke. 2021 · 2021

Earlier work this paper cites.

Frugalscore: Learning cheaper, lighter and faster evaluation metrics for automatic text generation

Eddine, Moussa Kamal, Guokan Shang, Antoine J.-P. Tixier, and Michalis Vazirgiannis. 2022 · 2022

Earlier work this paper cites.

Results of WMT22 metrics shared task: Stop using BLEU - neural metrics are better and more robust

Freitag, Markus, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George F. Foster, Alon Lavie, and André F. T. Martins. 2022 · 2022

Earlier work this paper cites.

Capturing failures of large language models via human cognitive biases

Jones, Erik and Jacob Steinhardt. 2022 · 2022

Earlier work this paper cites.

Maieutic prompting: Logically consistent reasoning with recursive explanations

Jung, Jaehun, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022 · 2022

Earlier work this paper cites.

Competition-level code generation with alphacode

Original

Li, Yujia, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022 · 2022

Earlier work this paper cites.

Teaching language models to support answers with verified quotes

Original

Menick, Jacob, Maja Trebacz, Vladimir Mikulik, John Aslanides, H. Francis Song, Martin J. Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese. 2022 · 2022

Earlier work this paper cites.

Adaptive testing and debugging of NLP models

Ribeiro, Marco Túlio and Scott M. Lundberg. 2022 · 2022

Earlier work this paper cites.

Self-critiquing models for assisting human evaluators

Original

Saunders, William, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022 · 2022

Earlier work this paper cites.

Bertscore is unfair: On social bias in language model-based metrics for text generation

Sun, Tianxiang, Junliang He, Xipeng Qiu, and Xuanjing Huang. 2022 · 2022

Earlier work this paper cites.

Language models with image descriptors are strong few-shot video-language learners

Wang, Zhenhailong, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, and Heng Ji. 2022 · 2022

Earlier work this paper cites.

Interpreting language models with contrastive explanations

Yin, Kayo and Graham Neubig. 2022 · 2022

Earlier work this paper cites.

OPT: open pre-trained transformer language models

Original

Zhang, Susan, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022 · 2022

Earlier work this paper cites.

Deconstructing NLG evaluation: Evaluation practices, assumptions, and their implications

Zhou, Kaitlyn, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, and Alexandra Olteanu. 2022 · 2022

Earlier work this paper cites.

Palm 2 technical report

Anil, Rohan, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. 2023 · 2023

Earlier work this paper cites.

Benchmarking foundation models with language-model-as-an-examiner

Bai, Yushi, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. 2023 · 2023

Earlier work this paper cites.

Can large language models be an alternative to human evaluations?

Chiang, David Cheng-Han and Hung-yi Lee. 2023a · 2023

Earlier work this paper cites.

A closer look into using large language models for automatic evaluation

Chiang, David Cheng-Han and Hung-yi Lee. 2023b · 2023

Earlier work this paper cites.

LM vs LM: detecting factual errors via cross examination

Cohen, Roi, May Hamri, Mor Geva, and Amir Globerson. 2023 · 2023

Earlier work this paper cites.

Are large language models reliable judges? A study on the factuality evaluation capabilities of llms

Original

Fu, Xue-Yong, Md. Tahmid Rahman Laskar, Cheng Chen, and Shashi Bhushan TN. 2023 · 2023

Earlier work this paper cites.

Human-like summarization evaluation with chatgpt

Original

Gao, Mingqi, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023 · 2023

Earlier work this paper cites.

Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text

Gehrmann, Sebastian, Elizabeth Clark, and Thibault Sellam. 2023 · 2023

Earlier work this paper cites.

Chatgpt outperforms crowd-workers for text-annotation tasks

Original

Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. 2023 · 2023

Earlier work this paper cites.

Coascore: Chain-of-aspects prompting for NLG evaluation

Original

Gong, Peiyuan and Jiaxin Mao. 2023 · 2023

Earlier work this paper cites.

ALLURE: auditing and improving llm-based evaluation of text using iterative in-context-learning

Original

Hasanbeig, Hosein, Hiteshi Sharma, Leo Betthauser, Felipe Vieira Frujeri, and Ida Momennejad. 2023 · 2023

Cited alongside, same era.

On the blind spots of model-based evaluation metrics for text generation

He, Tianxing, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James R. Glass, and Yulia Tsvetkov. 2023a · 2023

Cited alongside, same era.

On the blind spots of model-based evaluation metrics for text generation

He, Tianxing, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James R. Glass, and Yulia Tsvetkov. 2023b · 2023

Cited alongside, same era.

Decipherpref: Analyzing influential factors in human preference judgments via GPT-4

Hu, Yebowen, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, and Fei Liu. 2023 · 2023

Cited alongside, same era.

Multi-dimensional evaluation of text summarization with in-context learning

Jain, Sameer, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou. 2023 · 2023

Through the lens of core competency: Survey on evaluation of large language models

Original

Zhuang, Ziyu, Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Zixian Feng, Weinan Zhang, and Ting Liu. 2023 · 2023

Later among the works it cites.

Aligning human and LLM judgments: Insights from evalassist on task-specific evaluations and ai-assisted assessment strategy preferences

Original

Ashktorab, Zahra, Michael Desmond, Qian Pan, James M. Johnson, Martin Santillan Cooper, Elizabeth M. Daly, Rahul Nair, Tejaswini Pedapati, Swapnaja Achintalwar, and Werner Geyer. 2024 · 2024

Closest in time.

Llms instead of human judges? A large scale empirical study across 20 NLP evaluation tasks

Original

Bavaresco, Anna, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K. Surikuchi, Ece Takmaz, and Alberto Testoni. 2024 · 2024

Closest in time.

Compassjudger-1: All-in-one judge model helps model evaluation and evolution

Original