Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

Generalization through memorization: Nearest neighbor language models

Original

Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019 · 1911

Earlier work this paper cites.

Model-based feedback in the language modeling approach to information retrieval

Chengxiang Zhai and John Lafferty. 2001 · 2001

Earlier work this paper cites.

Longformer: The long-document transformer

Original

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020 · 2004

Earlier work this paper cites.

Leveraging passage retrieval with generative models for open domain question answering

Original

Gautier Izacard and Edouard Grave. 2020 · 2007

Earlier work this paper cites.

Adaptive relevance feedback in information retrieval

Yuanhua Lv and ChengXiang Zhai. 2009 · 2009

Earlier work this paper cites.

Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps

Original

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020 · 2011

Earlier work this paper cites.

The narrativeqa reading comprehension challenge

Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018 · 2018

Earlier work this paper cites.

Hotpotqa: A dataset for diverse, explainable multi-hop question answering

Original

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018 · 2018

Earlier work this paper cites.

Qmsum: A new benchmark for query-based multi-domain meeting summarization

Original

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021 · 2018

Earlier work this paper cites.

Retrieval augmented language model pre-training

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020 · 2020

Earlier work this paper cites.

Retrieval-augmented generation for knowledge-intensive nlp tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020 · 2020

Earlier work this paper cites.

A dataset of information-seeking questions and answers anchored in research papers

Original

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021 · 2021

Earlier work this paper cites.

Unsupervised dense information retrieval with contrastive learning

Original

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021 · 2021

Earlier work this paper cites.

Improving language models by retrieving from trillions of tokens

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022 · 2022

Earlier work this paper cites.

Longt5: Efficient text-to-text transformer for long sequences

Mandy Guo, Joshua Ainslie, David C Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022 · 2022

Earlier work this paper cites.

Scrolls: Standardized comparison over long language sequences

Original

Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022 · 2022

Earlier work this paper cites.

Musique: Multihop questions via single-hop question composition

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022 · 2022

Cited alongside, same era.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 · 2022

Cited alongside, same era.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Cited alongside, same era.

L-eval: Instituting standardized evaluation for long context language models

Original

Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023 · 2023

Cited alongside, same era.

Self-rag: Learning to retrieve, generate, and critique through self-reflection

Original

Claude 3.5 sonnet

Anthropic. 2024 · 2024

Closest in time.

Rq-rag: Learning to refine queries for retrieval augmented generation

Original

Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024 · 2024

Closest in time.

Unified active retrieval for retrieval augmented generation

Original

Qinyuan Cheng, Xiaonan Li, Shimin Li, Qin Zhu, Zhangyue Yin, Yunfan Shao, Linyang Li, Tianxiang Sun, Hang Yan, and Xipeng Qiu. 2024 · 2024

Closest in time.

Gemini pricing

Google. 2024 · 2024

Closest in time.

Ruler: What’s the real context size of your long-context language models?

Original

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024 · 2024

Closest in time.