Fetching the paper…

Reading the bibliography…

2024

In Defense of RAG in the Era of Long-Context Language Models

Yu, Tan, Xu, Anbang, Akkiraju, Rama

Original
PDF
Post on X

Understand

Overcoming the limited context limitations in early-generation LLMs, retrieval-augmented generation (RAG) has been a reliable solution for context-based answer generation in the past.

Recently, the emergence of long-context LLMs allows the models to incorporate much longer text sequences, making RAG less attractive.
Recent studies show that long-context LLMs significantly outperform RAG in long-context applications.
Unlike the existing works favoring the long-context LLM over RAG, we argue that the extremely long context in LLMs suffers from a diminished focus on relevant information and leads to potential degradation in answer quality.

In Defense of RAG in the Era of Long-Context Language Models · Around

Built on

Rethinking attention with performers
Original
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020 · 2009
Earlier work this paper cites.
Efficient transformers: A survey.(2020)
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020 · 2009
Earlier work this paper cites.
Retrieval augmented language model pre-training
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020 · 2020
Earlier work this paper cites.
Retrieval-augmented generation for knowledge-intensive nlp tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020 · 2020
Earlier work this paper cites.
Big bird: Transformers for longer sequences
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020 · 2020
Earlier work this paper cites.
Train short, test long: Attention with linear biases enables input length extrapolation
Original
Ofir Press, Noah A Smith, and Mike Lewis. 2021 · 2021
Earlier work this paper cites.
FlashAttention: Fast and memory-efficient exact attention with IO-awareness
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022 · 2022
Earlier work this paper cites.

Similar

A length-extrapolatable transformer
Original
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2022 · 2022
Cited alongside, same era.
Longbench: A bilingual, multitask benchmark for long context understanding
Original
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023 · 2023
Cited alongside, same era.
Extending context window of large language models via positional interpolation
Original
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023 · 2023
Cited alongside, same era.
Augmented language models: a survey
Original
Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. 2023 · 2023
Cited alongside, same era.

Then

Mistral large 2
Mistral AI. 2024 · 2024
Closest in time.
Claude 3.5 sonnet
Anthropic. 2024 · 2024
Closest in time.
FlashAttention-2: Faster attention with better parallelism and work partitioning
Tri Dao. 2024 · 2024
Closest in time.
Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach
Original
Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024 · 2024
Closest in time.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Original
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024 · 2024
Closest in time.
Grok-2 beta release

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

GPT-4 technical report

OpenAI. 2023 · 2023

Cited alongside, same era.

C-pack: Packaged resources to advance general chinese embedding

Original

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023 · 2023

Cited alongside, same era.

Phi-3 technical report: A highly capable language model locally on your phone

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024 · 2024

Cited alongside, same era.

Introducing llama 3.1: Our most capable models to date

Meta. 2024a

Cited in the paper.

Llama 3.1 models

Meta. 2024b

Cited in the paper.

xAI. 2024 · 2024

Closest in time.

∞ \infty bench: Extending long context evaluation beyond 100k tokens

Original

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024 · 2024

Closest in time.