"Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation

On faithfulness and factuality in abstractive summarization

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020 · 1919

Earlier work this paper cites.

Large language models can accurately predict searcher preferences

Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024 · 1940

Earlier work this paper cites.

Variations in relevance judgments and the measurement of retrieval effectiveness

Ellen M. Voorhees. 1998 · 1998

Earlier work this paper cites.

The probabilistic relevance framework: BM25 and beyond

Stephen E. Robertson and Hugo Zaragoza. 2009 · 2009

Earlier work this paper cites.

QuAC: Question answering in context

Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018 · 2018

Earlier work this paper cites.

BERT: pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Earlier work this paper cites.

Know what you don’t know: Unanswerable questions for SQuAD

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018 · 2018

Earlier work this paper cites.

Anserini: Reproducible ranking baselines using Lucene

Peilin Yang, Hui Fang, and Jimmy Lin. 2018 · 2018

Earlier work this paper cites.

Natural Questions: a benchmark for question answering research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019 · 2019

Earlier work this paper cites.

CoQA: A Conversational Question Answering Challenge

Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019 · 2019

Earlier work this paper cites.

Language models are few-shot learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020

Earlier work this paper cites.

Tydi QA: A benchmark for information-seeking question answering in typologically diverse languages

Jonathan H. Clark, Jennimaria Palomaki, Vitaly Nikolaev, Eunsol Choi, Dan Garrette, Michael Collins, and Tom Kwiatkowski. 2020 · 2020

Earlier work this paper cites.

Retrieval augmented language model pre-training

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020 · 2020

Earlier work this paper cites.

Dense passage retrieval for open-domain question answering

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020 · 2020

Earlier work this paper cites.

ColBERT: Efficient and effective passage search via contextualized late interaction over BERT

Omar Khattab and Matei Zaharia. 2020 · 2020

Earlier work this paper cites.

Retrieval-augmented generation for knowledge-intensive NLP tasks

Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020 · 2020

Earlier work this paper cites.

How context affects language models’ factual predictions

Fabio Petroni, Patrick S. H. Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020 · 2020

Earlier work this paper cites.

Editing factual knowledge in language models

Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021 · 2021

Earlier work this paper cites.

Leveraging passage retrieval with generative models for open domain question answering

Gautier Izacard and Edouard Grave. 2021 · 2021

Earlier work this paper cites.

Adversarial examples for evaluating math word problem solvers

Vivek Kumar, Rishabh Maheshwary, and Vikram Pudi. 2021 · 2021

Earlier work this paper cites.

The curious case of hallucinations in neural machine translation

Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021 · 2021

Cited alongside, same era.

Retrieval augmentation reduces hallucination in conversation

Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021 · 2021

Cited alongside, same era.

BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021 · 2021

Cited alongside, same era.

Improving language models by retrieving from trillions of tokens

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022 · 2022

Cited alongside, same era.

Retrieval-augmented generative question answering for event argument extraction

Xinya Du and Heng Ji. 2022 · 2022

GPT-4 technical report

Original

OpenAI. 2023 · 2023

Closest in time.

Large language models can be easily distracted by irrelevant context

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023 · 2023

Closest in time.

Llama 2: Open foundation and fine-tuned chat models

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023 · 2023

Closest in time.

Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions