Fetching the paper…

Reading the bibliography…

2023

Exploring the Practicality of Generative Retrieval on Dynamic Corpora

Kim, Chaeeun, Yoon, Soyoung, Lee, Hyunji et al.

Original
PDF
Post on X

Understand

Benchmarking the performance of information retrieval (IR) is mostly conducted with a fixed set of documents (static corpora).

However, in realistic scenarios, this is rarely the case and the documents to be retrieved are constantly updated and added.
In this paper, we focus on Generative Retrievals (GR), which apply autoregressive language models to IR problems, and explore their adaptability and robustness in dynamic scenarios.
We also conduct an extensive evaluation of computational and memory efficiency, crucial factors for real-world deployment of IR systems handling vast and ever-changing document collections.

Exploring the Practicality of Generative Retrieval on Dynamic Corpora · Around

Built on

Catastrophic interference in connectionist networks: The sequential learning problem
Michael McCloskey and Neal J. Cohen. 1989 · 1989
Earlier work this paper cites.
Temporal information retrieval
Nattiya Kanhabua and Avishek Anand. 2016 · 2016
Earlier work this paper cites.
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017 · 2017
Earlier work this paper cites.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Original
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017 · 2017
Earlier work this paper cites.
Ms marco: A human generated machine reading comprehension dataset
Original
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018 · 2018
Earlier work this paper cites.
End-to-end retrieval in continuous space
Original
Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. 2018 · 2018
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
Billion-scale similarity search with GPUs
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019 · 2019
Earlier work this paper cites.
Natural questions: A benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019 · 2019
Earlier work this paper cites.
Latent retrieval for weakly supervised open domain question answering
Original
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
From doc2query to doctttttquery
Rodrigo Nogueira and Jimmy Lin. 2019 · 2019
Earlier work this paper cites.
Language models as knowledge bases?
Original
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019 · 2019
Earlier work this paper cites.
Autoregressive entity retrieval
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020 · 2020
Earlier work this paper cites.
Scaling laws for neural language models
Original
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2020
Earlier work this paper cites.
Gshard: Scaling giant models with conditional computation and automatic sharding
Original
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020 · 2020
Earlier work this paper cites.
Approximate nearest neighbor negative contrastive learning for dense text retrieval
Original
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020 · 2020
Earlier work this paper cites.
Transformer feed-forward layers are key-value memories
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021 · 2021
Earlier work this paper cites.
Lora: Low-rank adaptation of large language models
Original
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021
Earlier work this paper cites.

Similar

Beyond distillation: Task-level mixture-of-experts for efficient inference
Original
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. 2021 · 2021
Cited alongside, same era.
A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques
Original
Jimmy Lin and Xueguang Ma. 2021 · 2021
Cited alongside, same era.
Learning passage impacts for inverted indexes
Original
Antonio Mallia, Omar Khattab, Nicola Tonellotto, and Torsten Suel. 2021 · 2021
Cited alongside, same era.
Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models
Original
Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. 2021 · 2021
Cited alongside, same era.

Then

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Original
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022 · 2022
Later among the works it cites.
Dynamicretriever: A pre-training model-based ir system with neither sparse nor dense index
Original
Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, and Ji-Rong Wen. 2022 · 2022
Later among the works it cites.
Bridging the gap between indexing and retrieval for differentiable search index with query generation
Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2022 · 2022
Later among the works it cites.
Composable sparse fine-tuning for cross-lingual transfer
Original
Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić. 2023 · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Kilt: a benchmark for knowledge intensive language tasks

Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021 · 2021

Cited alongside, same era.

Autoregressive search engines: Generating substrings as document identifiers

Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Wen tau Yih, Sebastian Riedel, and Fabio Petroni. 2022 · 2022

Cited alongside, same era.

Corpusbrain: Pre-train a generative retrieval model for knowledge-intensive language tasks

Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Yiqun Liu, Yixing Fan, and Xueqi Cheng. 2022 · 2022

Cited alongside, same era.

Knowledge neurons in pretrained transformers

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022 · 2022

Cited alongside, same era.

Flashattention: Fast and memory-efficient exact attention with io-awareness

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022 · 2022

Cited alongside, same era.

Time-aware language models as temporal knowledge bases

Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022 · 2022

Cited alongside, same era.

Simcse: Simple contrastive learning of sentence embeddings

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2022 · 2022

Cited alongside, same era.

Unsupervised dense information retrieval with contrastive learning

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022 · 2022

Cited alongside, same era.

Continual learning for generative retrieval over dynamic corpora

Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023 · 2023

Closest in time.

Flashattention-2: Faster attention with better parallelism and work partitioning

Original

Tri Dao. 2023 · 2023

Closest in time.

Longeval-retrieval: French-english dynamic test collection for continuous web search evaluation

Petra Galuščáková Romain Deveaud, Gabriela Gonzalez-Saez, Philippe Mulhem, Lorraine Goeuriot, Florina Piroi, and Martin Popel. 2023 · 2023

Closest in time.

Fast inference from transformers via speculative decoding

Original

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023 · 2023

Closest in time.

Dsi++: Updating transformer memory with new documents

Sanket Vaibhav Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Jinfeng Rao, Marc Najork, Emma Strubell, and Donald Metzler. 2023 · 2023

Closest in time.

How does generative retrieval scale to millions of passages?

Ronak Pradeep, Kai Hui, Jai Gupta, Adam D. Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q. Tran. 2023 · 2023

Closest in time.

Questions are all you need to train a dense passage retriever

Devendra Singh Sachan, Mike Lewis, Dani Yogatama, Luke Zettlemoyer, Joelle Pineau, and Manzil Zaheer. 2023 · 2023

Closest in time.

Learning to tokenize for generative retrieval

Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. 2023 · 2023

Closest in time.

A neural corpus indexer for document retrieval

Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Allen Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2023 · 2023

Closest in time.

Scalable and effective generative information retrieval

Hansi Zeng, Chen Luo, Bowen Jin, Sheikh Muhammad Sarwar, Tianxin Wei, and Hamed Zamani. 2023 · 2023

Closest in time.

Bridging the gap between indexing and retrieval for differentiable search index with query generation

Original

Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2023 · 2023

Closest in time.

Hallucination-free? assessing the reliability of leading ai legal research tools

Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, and Daniel E. Ho. 2024 · 2024

Closest in time.

Planning ahead in generative retrieval: Guiding autoregressive generation through simultaneous decoding

Hansi Zeng, Chen Luo, and Hamed Zamani. 2024 · 2024

Closest in time.