Fetching the paper…

Reading the bibliography…

2024

Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG

Jin, Bowen, Yoon, Jinsung, Han, Jiawei et al.

Original
PDF
Post on X

Understand

Retrieval-augmented generation (RAG) empowers large language models (LLMs) to utilize external knowledge sources.

The increasing capacity of LLMs to process longer input sequences opens up avenues for providing more retrieved information, to potentially enhance the quality of generated outputs.
It is plausible to assume that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance.
However, our empirical findings demonstrate that for many long-context LLMs, the quality of generated output initially improves first, but then subsequently declines as the number of retrieved passages increases.

Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG · Around

Built on

The probabilistic relevance framework: Bm25 and beyond
S. Robertson, H. Zaragoza, et al · 2009
Earlier work this paper cites.
Natural questions: a benchmark for question answering research
T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al · 2019
Earlier work this paper cites.
Dense passage retrieval for open-domain question answering
Original
V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih · 2020
Earlier work this paper cites.
Unsupervised dense information retrieval with contrastive learning
Original
G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave · 2021
Earlier work this paper cites.
Retrieving and reading: A comprehensive survey on open-domain question answering
Original
F. Zhu, W. Lei, C. Wang, J. Zheng, S. Poria, and T.-S. Chua · 2021
Earlier work this paper cites.
A survey on in-context learning
Original
Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui · 2022
Earlier work this paper cites.
Text embeddings by weakly-supervised contrastive pre-training
Original
L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al · 2022
Earlier work this paper cites.
Factuality challenges in the era of large language models
Original
I. Augenstein, T. Baldwin, M. Cha, T. Chakraborty, G. L. Ciampaglia, D. Corney, R. DiResta, E. Ferrara, S. Hale, A. Halevy, et al · 2023
Earlier work this paper cites.
Retrieval-augmented generation for large language models: A survey
Original
Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang · 2023
Earlier work this paper cites.
A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions
Original
L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al · 2023
Earlier work this paper cites.
Mistral 7b
Original
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al · 2023
Earlier work this paper cites.

Similar

Large language models can be easily distracted by irrelevant context
F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Schärli, and D. Zhou · 2023
Cited alongside, same era.
Retrieval meets long context large language models
Original
P. Xu, W. Ping, X. Wu, L. McAfee, C. Zhu, Z. Liu, S. Subramanian, E. Bakhturina, M. Shoeybi, and B. Catanzaro · 2023
Cited alongside, same era.
Instruction tuning for large language models: A survey
Original
S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu, et al · 2023
Cited alongside, same era.
A survey of large language models
Original
W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al · 2023
Cited alongside, same era.
Many-shot in-context learning
Original

Then

Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach
Original
Z. Li, C. Li, M. Zhang, Q. Mei, and M. Bendersky · 2024
Closest in time.
Ra-dit: Retrieval-augmented dual instruction tuning
X. V. Lin, X. Chen, M. Chen, W. Shi, M. Lomeli, R. James, P. Rodriguez, J. Kahn, G. Szilvasy, M. Lewis, et al · 2024
Closest in time.
Lost in the middle: How language models use long contexts
N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang · 2024
Closest in time.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Original
M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al · 2024
Closest in time.
Beyond the limits: A survey of techniques to extend the context length in large language models
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

R. Agarwal, A. Singh, L. M. Zhang, B. Bohnet, S. Chan, A. Anand, Z. Abbas, A. Nova, J. D. Co-Reyes, E. Chu, et al · 2024

Cited alongside, same era.

Self-rag: Learning to retrieve, generate, and critique through self-reflection

A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi · 2024

Cited alongside, same era.

Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation

J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu · 2024

Cited alongside, same era.

The power of noise: Redefining retrieval for rag systems

F. Cuconasu, G. Trappolini, F. Siciliano, S. Filice, C. Campagnano, Y. Maarek, N. Tonellotto, and F. Silvestri · 2024

Cited alongside, same era.

The llama 3 herd of models

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al · 2024

Cited alongside, same era.

Needle in a haystack - pressure testing llms, 2023

G. Kamradt · 2024

Cited alongside, same era.

Can long-context language models subsume retrieval, rag, sql, and more?

J. Lee, A. Chen, Z. Dai, D. Dua, D. S. Sachan, M. Boratko, Y. Luan, S. M. Arnold, V. Perot, S. Dalmia, et al · 2024

Cited alongside, same era.

Ruler: What’s the real context size of your long-context language models?

C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, and B. Ginsburg

Cited in the paper.

X. Wang, M. Salmani, P. Omidi, X. Ren, M. Rezagholizadeh, and A. Eshaghi · 2024

Closest in time.

Instructrag: Instructing retrieval-augmented generation with explicit denoising

Z. Wei, W.-L. Chen, and Y. Meng · 2024

Closest in time.

Making retrieval-augmented language models robust to irrelevant context

O. Yoran, T. Wolfson, O. Ram, and J. Berant · 2024

Closest in time.

Rankrag: Unifying context ranking with retrieval-augmented generation in llms

Y. Yu, W. Ping, Z. Liu, B. Wang, J. You, C. Zhang, M. Shoeybi, and B. Catanzaro · 2024

Closest in time.

Raft: Adapting language model to domain specific rag

T. Zhang, S. G. Patil, N. Jain, S. Shen, M. Zaharia, I. Stoica, and J. E. Gonzalez · 2024

Closest in time.

Dense text retrieval based on pretrained language models: A survey

W. X. Zhao, J. Liu, R. Ren, and J.-R. Wen · 2024

Closest in time.

A survey on efficient inference for large language models

Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y. Lou, L. Wang, Z. Yuan, X. Li, et al · 2024

Closest in time.