The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models

On the Evaluation of Machine-Generated Reports. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2024) . Washington, D.C., 1904–1915

James Mayfield, Eugene Yang, Dawn Lawrie, Sean MacAvaney, Paul McNamee, Douglas W. Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Kayi, Kate Sanders, Marc Mason, and Noah Hibbler. 2024 · 1915

Earlier work this paper cites.

Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1998) . Melbourne, Australia, 315–323

Ellen M. Voorhees. 1998 · 1998

Earlier work this paper cites.

How Reliable Are the Results of Large-Scale Information Retrieval Experiments?. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1998) . Melbourne, Australia, 307–314

Justin Zobel. 1998 · 1998

Earlier work this paper cites.

Evaluating Evaluation Measure Stability. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2000) . Athens, Greece, 33–40

Chris Buckley and Ellen M. Voorhees. 2000 · 2000

Earlier work this paper cites.

Evaluating Answers to Definition Questions. In Companion Volume of the Proceedings of HLT-NAACL 2003 — Short Papers . Edmonton, Canada, 109–111

Ellen M. Voorhees. 2003a · 2003

Earlier work this paper cites.

Overview of the TREC 2003 Question Answering Track. In Proceedings of the Twelfth Text REtrieval Conference (TREC 2003) . Gaithersburg, Maryland

Ellen M. Voorhees. 2003b · 2003

Earlier work this paper cites.

Retrieval Evaluation with Incomplete Information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2004) . Sheffield, United Kingdom, 25–32

Chris Buckley and Ellen M. Voorhees. 2004 · 2004

Earlier work this paper cites.

Evaluating Content Selection in Summarization: The Pyramid Method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004 . Boston, Massachusetts, 145–152

Ani Nenkova and Rebecca Passonneau. 2004 · 2004

Earlier work this paper cites.

Automatically Evaluating Answers to Definition Questions. In Proceedings of the 2005 Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005) . Vancouver, Canada, 931–938

Jimmy Lin and Dina Demner-Fushman. 2005 · 2005

Earlier work this paper cites.

Information Retrieval System Evaluation: Effort, Sensitivity, and Reliability. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005) . Salvador, Brazil, 162–169

Mark Sanderson and Justin Zobel. 2005 · 2005

Earlier work this paper cites.

Methods for Automatically Evaluating Answers to Complex Questions

Jimmy Lin and Dina Demner-Fushman. 2006a · 2006

Earlier work this paper cites.

Different Structures for Evaluating Answers to Complex Questions: Pyramids Won’t Topple, and Neither Will Human Assessors. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL 2007) . Prague, Czech Republic, 768–775

Hoa Trang Dang and Jimmy Lin. 2007 · 2007

Earlier work this paper cites.

Deconstructing Nuggets: The Stability and Reliability of Complex Question Answering Evaluation. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007) . Amsterdam, the Netherlands, 327–334

Jimmy Lin and Pengyi Zhang. 2007 · 2007

Earlier work this paper cites.

Information Retrieval Evaluation

Donna Harman. 2011 · 2011

Earlier work this paper cites.

IR System Evaluation using Nugget-based Test Collections. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (WSDM 2012) . Seattle, Washington, 393–402

Virgil Pavlu, Shahzad Rajput, Peter B. Golbus, and Javed A. Aslam. 2012 · 2012

Earlier work this paper cites.

REALM: Retrieval-Augmented Language Model Pre-training. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020) . 3929–3938

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020 · 2020

Earlier work this paper cites.

Generalization through Memorization: Nearest Neighbor Language Models. In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) . Addis Ababa, Ethiopia

Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020 · 2020

Earlier work this paper cites.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems 33 . 9459–9474

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020 · 2020

Cited alongside, same era.

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume . 874–880

Gautier Izacard and Edouard Grave. 2021 · 2021

Cited alongside, same era.

Improving Language Models by Retrieving from Trillions of Tokens. In Proceedings of the 39th International Conference on Machine Learning (ICML 2022) . Baltimore, Maryland, 2206–2240

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, et al · 2022

Cited alongside, same era.

Enabling Large Language Models to Generate Text with Citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing . Singapore, 6465–6488

Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023 · 2023

SFR-RAG: Towards Contextually Faithful LLMs

Original

Xuan-Phi Nguyen, Shrey Pandit, Senthil Purushwalkam, Austin Xu, Hailin Chen, Yifei Ming, Zixuan Ke, Silvio Savarese, Caiming Xong, and Shafiq Joty. 2024 · 2024

Later among the works it cites.

ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA Datasets with Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track . Miami, Florida, 1176–1206

Ronak Pradeep, Daniel Lee, Ali Mousavi, Jeffrey Pound, Yisi Sang, Jimmy Lin, Ihab Ilyas, Saloni Potdar, Mostafa Arefiyan, and Yunyao Li. 2024a · 2024

Later among the works it cites.

Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework

Original

Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024b · 2024

Later among the works it cites.

Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework

Original

Zackary Rackauckas, Arthur Câmara, and Jakub Zavrel. 2024 · 2024