Fetching the paper…

Reading the bibliography…

2022

SeDR: Segment Representation Learning for Long Documents Dense Retrieval

Chen, Junying, Chen, Qingcai, Li, Dongfang et al.

Original
PDF
Post on X

Understand

Recently, Dense Retrieval (DR) has become a promising solution to document retrieval, where document representations are used to perform effective and efficient semantic search.

However, DR remains challenging on long documents, due to the quadratic complexity of its Transformer-based encoder and the finite capacity of a low-dimension embedding.
Current DR models use suboptimal strategies such as truncating or splitting-and-pooling to long documents leading to poor utilization of whole document information.
In this work, to tackle this problem, we propose Segment representation learning for long documents Dense Retrieval (SeDR).

SeDR: Segment Representation Learning for Long Documents Dense Retrieval · Around

Built on

Large batch optimization for deep learning: Training bert in 76 minutes
Original
You, Y.; Li, J.; Reddi, S.; Hseu, J.; Kumar, S.; Bhojanapalli, S.; Song, X.; Demmel, J.; Keutzer, K.; and Hsieh, C.-J. 2019 · 1904
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019 · 1907
Earlier work this paper cites.
Blockwise self-attention for long document understanding
Original
Qiu, J.; Ma, H.; Levy, O.; Yih, S. W.-t.; Wang, S.; and Tang, J. 2019 · 1911
Earlier work this paper cites.
Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval
Robertson, S. E.; and Walker, S. 1994 · 1994
Earlier work this paper cites.
Reformer: The efficient transformer
Original
Kitaev, N.; Kaiser, Ł.; and Levskaya, A. 2020 · 2001
Earlier work this paper cites.
Overview of the TREC 2019 deep learning track
Original
Craswell, N.; Mitra, B.; Yilmaz, E.; Campos, D.; and Voorhees, E. M. 2020 · 2003
Earlier work this paper cites.
ETC: Encoding long and structured inputs in transformers
Original
Ainslie, J.; Ontanon, S.; Alberti, C.; Cvicek, V.; Fisher, Z.; Pham, P.; Ravula, A.; Sanghai, S.; Wang, Q.; and Yang, L. 2020 · 2004
Earlier work this paper cites.
Longformer: The long-document transformer
Original
Beltagy, I.; Peters, M. E.; and Cohan, A. 2020 · 2004
Earlier work this paper cites.
Complementing Lexical Retrieval with Semantic Residual Embedding
Original
Gao, L.; Dai, Z.; Fan, Z.; and Callan, J. 2020 · 2004
Earlier work this paper cites.
Dense passage retrieval for open-domain question answering
Original
Karpukhin, V.; Oğuz, B.; Min, S.; Lewis, P.; Wu, L.; Edunov, S.; Chen, D.; and Yih, W.-t. 2020 · 2004
Earlier work this paper cites.
Declutr: Deep contrastive learning for unsupervised textual representations
Original
Giorgi, J.; Nitski, O.; Wang, B.; and Bader, G. 2020 · 2006
Earlier work this paper cites.
Gmat: Global memory augmentation for transformers
Original
Gupta, A.; and Berant, J. 2020 · 2006
Earlier work this paper cites.
RepBERT: Contextualized text embeddings for first-stage retrieval
Original
Zhan, J.; Mao, J.; Liu, Y.; Zhang, M.; and Ma, S. 2020 · 2006
Earlier work this paper cites.

Similar

Approximate nearest neighbor negative contrastive learning for dense text retrieval
Original
Xiong, L.; Xiong, C.; Li, Y.; Tang, K.-F.; Liu, J.; Bennett, P.; Ahmed, J.; and Overwijk, A. 2020 · 2007
Cited alongside, same era.
Long document ranking with query-directed sparse transformer
Original
Jiang, J.-Y.; Xiong, C.; Lee, C.-J.; and Wang, W. 2020 · 2010
Cited alongside, same era.
RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering
Original
Qu, Y.; Ding, Y.; Liu, J.; Liu, K.; Ren, R.; Zhao, W. X.; Dong, D.; Wu, H.; and Wang, H. 2020 · 2010
Cited alongside, same era.
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Original
Nguyen, T.; Rosenberg, M.; Song, X.; Gao, J.; Tiwary, S.; Majumder, R.; and Deng, L. 2016 · 2016
Cited alongside, same era.

Then

Embedding-based retrieval in facebook search
Huang, J.-T.; Sharma, A.; Sun, S.; Xia, L.; Zhang, D.; Pronin, P.; Padmanabhan, J.; Ottaviano, G.; and Yang, L. 2020 · 2020
Later among the works it cites.
Colbert: Efficient and effective passage search via contextualized late interaction over bert
Khattab, O.; and Zaharia, M. 2020 · 2020
Later among the works it cites.
Big bird: Transformers for longer sequences
Zaheer, M.; Guruganesh, G.; Dubey, K. A.; Ainslie, J.; Alberti, C.; Ontanon, S.; Pham, P.; Ravula, A.; Wang, Q.; Yang, L.; et al. 2020 · 2020
Later among the works it cites.
Simcse: Simple contrastive learning of sentence embeddings
Original
Gao, T.; Yao, X.; and Chen, D. 2021 · 2021
Later among the works it cites.
Sparse, dense, and attentional representations for text retrieval
Luan, Y.; Eisenstein, J.; Toutanova, K.; and Collins, M. 2021 · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Attention is all you need

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017 · 2017

Cited alongside, same era.

Bert: Pre-training of deep bidirectional transformers for language understanding

Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018 · 2018

Cited alongside, same era.

Representation learning with contrastive predictive coding

Oord, A. v. d.; Li, Y.; and Vinyals, O. 2018 · 2018

Cited alongside, same era.

Anserini: Reproducible ranking baselines using Lucene

Yang, P.; Fang, H.; and Lin, J. 2018 · 2018

Cited alongside, same era.

Deeper text understanding for IR with contextual neural language modeling

Dai, Z.; and Callan, J. 2019 · 2019

Cited alongside, same era.

Billion-scale similarity search with gpus

Johnson, J.; Douze, M.; and Jégou, H. 2019 · 2019

Cited alongside, same era.

From doc2query to docTTTTTquery

Nogueira, R.; Lin, J.; and Epistemic, A. 2019 · 2019

Cited alongside, same era.

A simple framework for contrastive learning of visual representations

Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020 · 2020

Cited alongside, same era.

Matching-oriented Product Quantization For Ad-hoc Retrieval

Xiao, S.; Liu, Z.; Shao, Y.; Lian, D.; and Xie, X. 2021 · 2021

Later among the works it cites.

Optimizing dense retrieval model training with hard negatives

Zhan, J.; Mao, J.; Liu, Y.; Guo, J.; Zhang, M.; and Ma, S. 2021 · 2021

Later among the works it cites.

Leveraging Multi-view Inter-passage Interactions for Neural Document Ranking

Fu, C.; Hu, E.; Feng, L.; Dou, Z.; Jia, Y.; Chen, L.; Yu, F.; and Cao, Z. 2022 · 2022

Closest in time.

Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval

Xiao, S.; Liu, Z.; Han, W.; Zhang, J.; Shao, Y.; Lian, D.; Li, C.; Sun, H.; Deng, D.; Zhang, L.; et al. 2022 · 2022

Closest in time.

Multi-View Document Representation Learning for Open-Domain Dense Retrieval

Zhang, S.; Liang, Y.; Gong, M.; Jiang, D.; and Duan, N. 2022 · 2022

Closest in time.

Socialformer: Social Network Inspired Long Document Modeling for Document Ranking

Zhou, Y.; Dou, Z.; Yuan, H.; and Ma, Z. 2022 · 2022

Closest in time.

Local self-attention over long text for efficient document retrieval

Hofstätter, S.; Zamani, H.; Mitra, B.; Craswell, N.; and Hanbury, A. 2020a · 2024

Closest in time.

Local self-attention over long text for efficient document retrieval

Hofstätter, S.; Zamani, H.; Mitra, B.; Craswell, N.; and Hanbury, A. 2020b · 2024

Closest in time.