Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges

Retrieval Augmented Language Model Pre-Training

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020 · 2020

Earlier work this paper cites.

Generalization through Memorization: Nearest Neighbor Language Models

Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020 · 2020

Earlier work this paper cites.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020 · 2020

Earlier work this paper cites.

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

Gautier Izacard and Edouard Grave. 2021 · 2021

Earlier work this paper cites.

Improving Language Models by Retrieving from Trillions of Tokens

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022 · 2022

Earlier work this paper cites.

LangChain

Harrison Chase. 2022 · 2022

Earlier work this paper cites.

Overview of the TREC 2022 Deep Learning Track

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2022 · 2022

Earlier work this paper cites.

TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants

Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffrey Dalton, and Leif Azzopardi. 2024 · 2023

Earlier work this paper cites.

Faithfulness Tests for Natural Language Explanations

Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. 2023 · 2023

Earlier work this paper cites.

Overview of the TREC 2023 Deep Learning Track

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2023 · 2023

Earlier work this paper cites.

Enabling Large Language Models to Generate Text with Citations

Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023 · 2023

Earlier work this paper cites.

Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model

Qi Jia, Siyu Ren, Yizhu Liu, and Kenny Q. Zhu. 2023 · 2023

Cited alongside, same era.

SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization

Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander R. Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023 · 2023

Cited alongside, same era.

Evaluating Verifiability in Generative Search Engines

Nelson Liu, Tianyi Zhang, and Percy Liang. 2023 · 2023

Cited alongside, same era.

Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web

Microsoft. 2023 · 2023

Cited alongside, same era.

AToMiC: An Image/Text Retrieval Test Collection to Support Multimedia Content Creation

Jheng-Hong Yang, Carlos Lassance, Rafael Sampaio De Rezende, Krishna Srinivasan, Miriam Redi, Stéphane Clinchant, and Jimmy Lin. 2023 · 2023

Cited alongside, same era.

Hello GPT-4o

OpenAI. 2024 · 2024

Later among the works it cites.

Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track

Original

Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024a · 2024

Later among the works it cites.

Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework

Original

Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024b · 2024

Later among the works it cites.

Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents

Original

Corby Rosset, Ho-Lam Chung, Guanghui Qin, Ethan C. Chau, Zhuo Feng, Ahmed Awadallah, Jennifer Neville, and Nikhil Rao. 2024 · 2024

Later among the works it cites.

RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation