Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets

T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating text generation with BERT.” [Online]. Available: http://arxiv.org/abs/1904.09675

Original

1904

Earlier work this paper cites.

N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using siamese BERT-networks.” [Online]. Available: http://arxiv.org/abs/1908.10084

Original

1908

Earlier work this paper cites.

P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” Apr. 2021, arXiv:2005.11401. [Online]. Available: http://arxiv.org/abs/2005.11401

Original

2005

Earlier work this paper cites.

B. Kitchenham and S. Charters, “Guidelines for performing systematic literature reviews in software engineering,” vol. 2, 2007

2007

Earlier work this paper cites.

Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, L. Wang, A. Luu, W. Bi, F. Shi, and S. Shi, “Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models,” ArXiv , Sep. 2023

2023

Earlier work this paper cites.

Zeta Alpha Vector, “Ragelo,” 2023. [Online]. Available: https://github.com/zetaalphavector/RAGElo

2023

Cited alongside, same era.

S. Kukreja, T. Kumar, V. Bharate, A. Purohit, A. Dasgupta, and D. Guha, “Performance evaluation of vector embeddings with retrieval-augmented generation,” in 2024 9th International Conference on Computer and Communication Systems (ICCCS) , 2024, pp. 333–340. [Online]. Available: https://ieeexplore.ieee.org/document/10603291

2024

Cited alongside, same era.

H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, “Evaluation of retrieval-augmented generation: A survey,” 2024. [Online]. Available: http://arxiv.org/abs/2405.07437

Original

2024

Cited alongside, same era.

S. Knollmeyer, O. Caymazer, L. Koval, M. Akmal, S. Asif, S. Mathias, and D. Großmann, “Benchmarking of retrieval augmented generation: A comprehensive systematic literature review on evaluation dimensions, evaluation metrics and datasets,” in Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - KMIS , INSTICC. SciTePress, 2024, pp. 137–148

S. Khaled, E. H. Mohamed, and W. Medhat, “Evaluating large language models for arabic sentiment analysis: A comparative study using retrieval-augmented generation,” Procedia Computer Science , vol. 244, pp. 363–370, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1877050924030114

2024

Later among the works it cites.

X. Yu, H. Cheng, X. Liu, D. Roth, and J. Gao, “ReEval: Automatic hallucination evaluation for retrieval-augmented large language models via transferable adversarial attacks,” in Findings of the Association for Computational Linguistics: NAACL 2024 , K. Duh, H. Gomez, and S. Bethard, Eds. Association for Computational Linguistics, 2024, pp. 1333–1351. [Online]. Available: https://aclanthology.org/2024.findings-naacl.85

2024

Later among the works it cites.

C. Lang, R. Schneider, and N. D. T. Tu, “Automatic question answering for the linguistic domain – an evaluation of LLM knowledge base extension with RAG,” in Natural Language Processing and Information Systems , A. Rapp, L. Di Caro, F. Meziane, and V. Sugumaran, Eds. Springer Nature Switzerland, 2024, pp. 161–171

2024