Fetching the paper…

Reading the bibliography…

2021

SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking

Formal, Thibault, Piwowarski, Benjamin, Clinchant, Stéphane

Original
PDF
Post on X

Understand

In neural Information Retrieval, ongoing research is directed towards improving the first retriever in ranking pipelines.

Learning dense embeddings to conduct retrieval using efficient approximate nearest neighbors methods has proven to work well.
Meanwhile, there has been a growing interest in learning sparse representations for documents and queries, that could inherit from the desirable properties of bag-of-words models such as the exact matching of terms and the efficiency of inverted indexes.
In this work, we present a new first-stage ranker based on explicit sparsity regularization and a log-saturation effect on term weights, leading to highly sparse representations and competitive results with respect to state-of-the-art dense and sparse methods.

SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking · Around

Built on

Passage Re-ranking with BERT
Original
Rodrigo Nogueira and Kyunghyun Cho. 2019 · 1901
Earlier work this paper cites.
Document Expansion by Query Prediction
Original
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019 · 1904
Earlier work this paper cites.
Context-Aware Document Term Weighting for Ad-Hoc Search
Zhuyun Dai and Jamie Callan. 2020a · 1907
Earlier work this paper cites.
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics
Original
Nils Reimers and Iryna Gurevych. 2019 · 1908
Earlier work this paper cites.
Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
Original
Zhuyun Dai and Jamie Callan. 2019 · 1910
Earlier work this paper cites.
HuggingFace’s Transformers: State-of-the-art Natural Language Processing
Original
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020 · 1910
Earlier work this paper cites.
REALM: Retrieval-Augmented Language Model Pre-Training
Original
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020 · 2002
Earlier work this paper cites.
A Formal Study of Information Retrieval Heuristics. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Sheffield, United Kingdom) (SIGIR ’04) . Association for Computing Machinery, New York, NY, USA, 49–56
Hui Fang, Tao Tao, and ChengXiang Zhai. 2004 · 2004
Earlier work this paper cites.
Dense Passage Retrieval for Open-Domain Question Answering
Original
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020 · 2004
Earlier work this paper cites.
Minimizing FLOPs to Learn Efficient Sparse Representations
Original
Biswajit Paria, Chih-Kuan Yeh, Ian E. H. Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. 2020 · 2004
Earlier work this paper cites.

Similar

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Original
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020 · 2007
Cited alongside, same era.
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
Original
Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2020 · 2009
Cited alongside, same era.
SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
Original
Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. 2020 · 2010
Cited alongside, same era.
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
Original
Yingqi Qu Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2020 · 2010
Cited alongside, same era.

Then

From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (Torino, Italy) (CIKM ’18) . Association for Computing Machinery, New York, NY, USA, 497–506
Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik Learned-Miller, and Jaap Kamps. 2018 · 2018
Later among the works it cites.
From doc2query to docTTTTTquery
Rodrigo Nogueira and Jimmy Lin. 2019 · 2019
Later among the works it cites.
PyTorch: An Imperative Style, High-Performance Deep Learning Library.. In NeurIPS
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019
Later among the works it cites.
Critically Examining the “Neural Hype”
Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019 · 2019
Later among the works it cites.
Overview of the trec 2019 deep learning track
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation

Original

Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020 · 2010

Cited alongside, same era.

Distilling Dense Representations for Ranking using Tightly-Coupled Teachers

Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2020 · 2010

Cited alongside, same era.

Numba: A llvm-based python jit compiler. In Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC . 1–6

Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. 2015 · 2015

Cited alongside, same era.

Efficient and Accurate Non-Metric k-NN Search with Applications to Text Matching

Leonid Boytsov. 2018 · 2018

Cited alongside, same era.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Cited alongside, same era.

Context-Aware Term Weighting For First Stage Passage Retrieval

Zhuyun Dai and Jamie Callan. 2020b

Cited in the paper.

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020 · 2020

Later among the works it cites.

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20) . Association for Computing Machinery, New York, NY, USA, 39–48

Omar Khattab and Matei Zaharia. 2020 · 2020

Later among the works it cites.

Expansion via Prediction of Importance with Contextualization

Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020 · 2020

Later among the works it cites.

Approximate Nearest Neighbor Search and Lightweight Dense Vector Reranking in Multi-Stage Retrieval Architectures. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval . 97–100

Zhengkai Tu, Wei Yang, Zihang Fu, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2020 · 2020

Later among the works it cites.

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In International Conference on Learning Representations

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwikj. 2021 · 2021

Closest in time.