Fetching the paper…

Reading the bibliography…

2024

A Theory for Token-Level Harmonization in Retrieval-Augmented Generation

Xu, Shicheng, Pang, Liang, Shen, Huawei et al.

Original
PDF
Post on X

Understand

Retrieval-augmented generation (RAG) utilizes retrieved texts to enhance large language models (LLMs).

Studies show that while RAG provides valuable external information (benefit), it may also mislead LLMs (detriment) with noisy or incorrect retrieved texts.
Although many existing methods attempt to preserve benefit and avoid detriment, they lack a theoretical explanation for RAG.
The benefit and detriment in the next token prediction of RAG remain a black box that cannot be quantified or compared in an explainable manner, so existing methods are data-driven, need additional utility evaluators or post-hoc.

A Theory for Token-Level Harmonization in Retrieval-Augmented Generation · Around

Built on

Theoretical foundation of potential functions method in pattern recognition
Mark A Aizerman, Emmanuil M Braverman, and Lev I Rozonoer · 1964
Earlier work this paper cites.
A tutorial on hidden markov models and selected applications in speech recognition
Lawrence R Rabiner · 1989
Earlier work this paper cites.
A neural probabilistic language model
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent · 2000
Earlier work this paper cites.
Latent dirichlet allocation
David M Blei, Andrew Y Ng, and Michael I Jordan · 2003
Earlier work this paper cites.
The wikitext long term dependency language modeling dataset
Stephen Merity · 2016
Earlier work this paper cites.
Squad: 100,000+ questions for machine comprehension of text
Original
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang · 2016
Earlier work this paper cites.
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension
Original
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
A broad-coverage challenge corpus for sentence understanding through inference
Original
Adina Williams, Nikita Nangia, and Samuel R Bowman · 2017
Earlier work this paper cites.
Wizard of wikipedia: Knowledge-powered conversational agents
Original
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston · 2018
Earlier work this paper cites.
T-rex: A large scale alignment of natural language with knowledge base triples
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl · 2018
Earlier work this paper cites.
Mapping language to code in programmatic context
Original
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer · 2018
Earlier work this paper cites.
Eli5: Long form question answering
Original
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli · 2019
Earlier work this paper cites.
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer · 2019
Earlier work this paper cites.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019
Earlier work this paper cites.
Bert rediscovers the classical nlp pipeline
Original
Ian Tenney, Dipanjan Das, and Ellie Pavlick · 2019
Earlier work this paper cites.
Dense passage retrieval for open-domain question answering
Original
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih · 2020
Earlier work this paper cites.
Uncertainty estimation in autoregressive structured prediction
Original
Andrey Malinin and Mark Gales · 2020
Earlier work this paper cites.

Similar

Kilt: a benchmark for knowledge intensive language tasks
Original
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al · 2020
Cited alongside, same era.
Codebleu: a method for automatic evaluation of code synthesis
Original
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma · 2020
Cited alongside, same era.
Colbertv2: Effective and efficient retrieval via lightweight late interaction
Original
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia · 2021
Cited alongside, same era.
An explanation of in-context learning as implicit bayesian inference
Original
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma · 2021
Cited alongside, same era.

Then

How do transformers learn topic structure: Towards a mechanistic understanding
Yuchen Li, Yuanzhi Li, and Andrej Risteski · 2023
Later among the works it cites.
In-context retrieval-augmented language models
Original
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham · 2023
Later among the works it cites.
Investigating the factual knowledge boundary of large language models with retrieval augmentation
Original
Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang · 2023
Later among the works it cites.
Toolformer: Language models can teach themselves to use tools
Original
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom · 2023
Later among the works it cites.
Replug: Retrieval-augmented black-box language models

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

What learning algorithm is in-context learning? investigations with linear models

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou · 2022

Cited alongside, same era.

Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei · 2022

Cited alongside, same era.

The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention

Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber · 2022

Cited alongside, same era.

Towards collaborative neural-symbolic graph semantic parsing via uncertainty

Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang · 2022

Cited alongside, same era.

When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi · 2022

Cited alongside, same era.

Confident adaptive language modeling

Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler · 2022

Cited alongside, same era.

Self-rag: Learning to retrieve, generate, and critique through self-reflection

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi · 2023

Cited alongside, same era.

Dola: Decoding by contrasting layers improves factuality in large language models

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He · 2023

Cited alongside, same era.

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih · 2023

Later among the works it cites.

Transformers learn in-context by gradient descent

Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov · 2023

Later among the works it cites.

Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge conflicts

Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su · 2023

Later among the works it cites.

Search-in-the-chain: Towards the accurate, credible and traceable content generation for complex knowledge-intensive tasks

Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-seng Chua · 2023

Later among the works it cites.

What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization

Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang · 2023

Later among the works it cites.

Inside: Llms’ internal states retain the power of hallucination detection

Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye · 2024

Closest in time.

Retrieve only when it needs: Adaptive retrieval augmentation for hallucination mitigation in large language models

Original

Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng · 2024

Closest in time.

Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models

Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Qiuxia Li, and Jun Zhao · 2024

Closest in time.

Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning

Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang · 2024

Closest in time.

Unsupervised information refinement training of large language models for retrieval-augmented generation

Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, and Jie Zhou · 2024

Closest in time.

Corrective retrieval augmented generation

Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling · 2024

Closest in time.

Making retrieval-augmented language models robust to irrelevant context

Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant · 2024

Closest in time.

Cross-modal safety mechanism transfer in large vision-language models

Shicheng Xu, Liang Pang, Yunchang Zhu, Huawei Shen, and Xueqi Cheng · 2025

Closest in time.