LongEmbed: Extending Embedding Models for Long Context Retrieval

Randomized positional encodings boost length generalization of transformers

Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. 2023 · 1903

Earlier work this paper cites.

Indexing by latent semantic analysis

Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. 1990 · 1990

Earlier work this paper cites.

Efficient estimation of word representations in vector space

Original

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 · 2013

Earlier work this paper cites.

Ms marco: A human-generated machine reading comprehension dataset

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016 · 2016

Earlier work this paper cites.

Neural tangent kernel: Convergence and generalization in neural networks

Arthur Jacot, Franck Gabriel, and Clément Hongler. 2018 · 2018

Earlier work this paper cites.

The NarrativeQA reading comprehension challenge

Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018 · 2018

Earlier work this paper cites.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Natural questions: A benchmark for question answering research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019 · 2019

Earlier work this paper cites.

Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020 · 2020

Earlier work this paper cites.

Dense passage retrieval for open-domain question answering

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020 · 2020

Earlier work this paper cites.

Simcse: Simple contrastive learning of sentence embeddings

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021 · 2021

Earlier work this paper cites.

Towards unsupervised dense information retrieval with contrastive learning

Original

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021 · 2021

Earlier work this paper cites.

Understanding attention scaling from the perspective of entropy invariance

Jianlin Su. 2021 · 2021

Earlier work this paper cites.

Roformer: Enhanced transformer with rotary position embedding

Original

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021 · 2021

Earlier work this paper cites.

BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021 · 2021

Earlier work this paper cites.

QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021 · 2021

Earlier work this paper cites.

Summscreen: A dataset for abstractive screenplay summarization

Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022 · 2022

Earlier work this paper cites.

Overcoming a theoretical limitation of self-attention

David Chiang and Peter Cholak. 2022 · 2022

Cited alongside, same era.

xformers: A modular and hackable transformer modelling library

Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. 2022 · 2022

Cited alongside, same era.

Text and code embeddings by contrastive pre-training

Original

Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. 2022 · 2022

Cited alongside, same era.

Large dual encoders are generalizable retrievers

Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al. 2022 · 2022

Cited alongside, same era.

SCROLLS: Standardized CompaRison over long language sequences

Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022 · 2022

Llama: Open and efficient foundation language models

Original

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 · 2023

Later among the works it cites.

C-pack: Packaged resources to advance general chinese embedding

Original

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. 2023 · 2023

Later among the works it cites.

Effective long-context scaling of foundation models

Original

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2023 · 2023

Later among the works it cites.

Pose: Efficient context window extension of llms via positional skip-wise training

Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2023 · 2023

Later among the works it cites.