Fetching the paper…
Reading the bibliography…
This study investigates the existence of positional biases in Transformer-based models for text representation learning, particularly in the context of web document retrieval.
Longformer: The Long-Document Transformer
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020 · 2004
Earlier work this paper cites.
PARADE: passage representation aggregation for document reranking
Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020 · 2008
Earlier work this paper cites.
MS MARCO: A human generated machine reading comprehension dataset
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016 · 2016
Earlier work this paper cites.
WikiHow: A Large Scale Text Summarization Dataset
Mahnaz Koupaee and William Yang Wang. 2018 · 2018
Earlier work this paper cites.
Deeper text understanding for IR with contextual neural language modeling
Zhuyun Dai and Jamie Callan. 2019 · 2019
Earlier work this paper cites.
Latent Retrieval for Weakly Supervised Open Domain Question Answering
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
Dense Passage Retrieval for Open-Domain Question Answering
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020 · 2020
Earlier work this paper cites.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020
Earlier work this paper cites.
Big Bird: Transformers for Longer Sequences
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020 · 2020
Earlier work this paper cites.
Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. 2021 · 2021
Earlier work this paper cites.
Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decode
Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021 · 2021
Earlier work this paper cites.
BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021 · 2021
Earlier work this paper cites.
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021 · 2021
Earlier work this paper cites.
Optimizing Dense Retrieval Model Training with Hard Negatives
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021 · 2021
Earlier work this paper cites.
Leonid Boytsov, Tianyi Lin, Fangwei Gao, Yutian Zhao, Jeffrey Huang, and Eric Nyberg. 2022 · 2022
Cited alongside, same era.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022 · 2022
Cited alongside, same era.
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
Luyu Gao and Jamie Callan. 2022 · 2022
Cited alongside, same era.
Unsupervised Dense Information Retrieval with Contrastive Learning
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022 · 2022
Cited alongside, same era.
Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, and Xueqi Cheng. 2022 · 2022
Cited alongside, same era.
Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023 · 2023
Later among the works it cites.
Learning to Compress Prompts with Gist Tokens
Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. 2023 · 2023
Later among the works it cites.
MTEB: massive text embedding benchmark
Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023 · 2023
Later among the works it cites.
LLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. 2022 · 2022
Cited alongside, same era.
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Ofir Press, Noah A. Smith, and Mike Lewis. 2022 · 2022
Cited alongside, same era.
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022 · 2022
Cited alongside, same era.
RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder
Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022 · 2022
Cited alongside, same era.
Adapting Language Models to Compress Contexts
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023 · 2023
Cited alongside, same era.
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Tri Dao. 2023 · 2023
Cited alongside, same era.
Overview of the TREC 2023 NeuCLIR Track
Dawn J. Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, and Eugene Yang. 2024 · 2023
Cited alongside, same era.
Kexin Wang, Nils Reimers, and Iryna Gurevych. 2023 · 2023
Later among the works it cites.
Effective Long-Context Scaling of Foundation Models
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. 2023 · 2023
Later among the works it cites.
Longtriever: a Pre-trained Long Text Encoder for Dense Document Retrieval
Junhan Yang, Zheng Liu, Chaozhuo Li, Guangzhong Sun, and Xing Xie. 2023 · 2023
Later among the works it cites.
Language Models are Universal Embedders
Xin Zhang, Zehan Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, and Min Zhang. 2023 · 2023
Later among the works it cites.
User Embedding Model for Personalized Language Prompting
Sumanth Doddapaneni, Krishna Sayana, Ambarish Jash, Sukhdeep S. Sodhi, and Dima Kuzmin. 2024 · 2024
Closest in time.
Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval
Guangyuan Ma, Xing Wu, Zijia Lin, and Songlin Hu. 2024 · 2024
Closest in time.
YaRN: Efficient Context Window Extension of Large Language Models
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024 · 2024
Closest in time.
Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT
Jon Saad-Falcon, Daniel Y. Fu, Simran Arora, Neel Guha, and Christopher Ré. 2024 · 2024
Closest in time.
RoFormer: Enhanced transformer with Rotary Position Embedding
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024 · 2024
Closest in time.