The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches

T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, BERTScore: Evaluating Text Generation with BERT, 2020. URL: http://arxiv.org/abs/1904.09675 . doi: 10.48550/arXiv.1904.09675 , arXiv:1904.09675 [cs]

Original

1904

Earlier work this paper cites.

Bleu: a Method for Automatic Evaluation of Machine Translation,

K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, · 2002

Earlier work this paper cites.

ROUGE: A Package for Automatic Evaluation of Summaries,

C.-Y. Lin, · 2004

Earlier work this paper cites.

T. Sellam, D. Das, A. P. Parikh, BLEURT: Learning Robust Metrics for Text Generation, 2020. URL: http://arxiv.org/abs/2004.04696 . doi: 10.48550/arXiv.2004.04696 , arXiv:2004.04696 [cs]

Original

2004

Earlier work this paper cites.

METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments,

S. Banerjee, A. Lavie, · 2005

Earlier work this paper cites.

D. Card, P. Henderson, U. Khandelwal, R. Jia, K. Mahowald, D. Jurafsky, With Little Power Comes Great Responsibility, 2020. URL: http://arxiv.org/abs/2010.06595 , arXiv:2010.06595 [cs]

Original

2010

Earlier work this paper cites.

P. Armstrong, Bloom’s Taxonomy, 2010. URL: https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/

2010

Earlier work this paper cites.

Best practices for the human evaluation of automatically generated text,

C. van der Lee, A. Gatt, E. Van Miltenburg, S. Wubben, E. Krahmer, · 2019

Earlier work this paper cites.

Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review,

A. Abd-Alrazaq, Z. Safi, M. Alajlani, J. Warren, M. Househ, K. Denecke, · 2020

Earlier work this paper cites.

Human evaluation of automatically generated text: Current trends and best practice guidelines,

C. van der Lee, A. Gatt, E. van Miltenburg, E. Krahmer, · 2020

Earlier work this paper cites.

“This is a Problem, Don’t You Agree?” Framing and Bias in Human Evaluation for Natural Language Generation,

S. Schoch, D. Yang, Y. Ji, · 2020

Cited alongside, same era.

Algorithm Inspection for Chatbot Performance Evaluation,

V. Vijayaraghavan, J. B. Cooper, R. L. J., · 2020

Cited alongside, same era.

E. Clark, T. August, S. Serrano, N. Haduong, S. Gururangan, N. A. Smith, All That’s ’Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text, 2021. URL: http://arxiv.org/abs/2107.00061 , arXiv:2107.00061 [cs]

Original

2021

Cited alongside, same era.

What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think,

D. M. Howcroft, V. Rieser, · 2021

Cited alongside, same era.

B. Srivastava, K. Lakkaraju, T. Koppel, V. Narayanan, A. Kundu, S. Joshi, Evaluating Chatbots to Promote Users’ Trust – Practices and Open Problems, 2023. URL: http://arxiv.org/abs/2309.05680 , arXiv:2309.05680 [cs]

Original

P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, Y. Koreeda, Holistic Evaluation of Language Models, 2023. URL: http://arxiv.org/abs/2211.09110 . doi: 10.48550/arXiv.2211.09110 , arXiv:2211.09110 [cs]

Original

2023

Later among the works it cites.

H. Nori, Y. T. Lee, S. Zhang, D. Carignan, R. Edgar, N. Fusi, N. King, J. Larson, Y. Li, W. Liu, R. Luo, S. M. McKinney, R. O. Ness, H. Poon, T. Qin, N. Usuyama, C. White, E. Horvitz, Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine, 2023. URL: http://arxiv.org/abs/2311.16452 . doi: 10.48550/arXiv.2311.16452 , arXiv:2311.16452 [cs]

Original

2023

Later among the works it cites.

Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, Retrieval-Augmented Generation for Large Language Models: A Survey, 2023. URL: http://arxiv.org/abs/2312.10997 . doi: 10.48550/arXiv.2312.10997 , arXiv:2312.10997 [cs]

Original

2023

Later among the works it cites.