Fetching the paper…

Reading the bibliography…

2021

Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline

Gao, Luyu, Dai, Zhuyun, Callan, Jamie

Original
PDF
Post on X

Understand

Pre-trained deep language models~(LM) have advanced the state-of-the-art of text retrieval.

Rerankers fine-tuned from deep LM estimates candidate relevance based on rich contextualized matching signals.
Meanwhile, deep LMs can also be leveraged to improve search index, building retrievers with better recall.
One would expect a straightforward combination of both in a pipeline to have additive performance gain.

Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline · Around

Built on

Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) 2
2006
Earlier work this paper cites.
Campos, D.F., Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., Deng, L., Mitra, B.: Ms marco: A human generated machine reading comprehension dataset. ArXiv abs/1611.09268
Original
2016
Earlier work this paper cites.
Yang, P., Fang, H., Lin, J.: Anserini: Enabling the use of lucene for information retrieval research. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (2017)
2017
Earlier work this paper cites.
Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. ArXiv abs/1802.05365
Original
2018
Earlier work this paper cites.
Wu, Z., Xiong, Y., Yu, S., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 3733–3742 (2018)
2018
Earlier work this paper cites.
Dai, Z., Callan, J.: Deeper text understanding for ir with contextual neural language modeling. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (2019)
2019
Earlier work this paper cites.

Similar

Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2019)
2019
Cited alongside, same era.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. ArXiv abs/1907.11692
Original
2019
Cited alongside, same era.
Nogueira, R., Cho, K.: Passage re-ranking with bert. ArXiv abs/1901.04085
Original
2019
Cited alongside, same era.
Nogueira, R., Yang, W., Cho, K., Lin, J.: Multi-stage document ranking with bert. ArXiv abs/1910.14424
Original
2019
Cited alongside, same era.
Nogueira, R., Yang, W., Lin, J., Cho, K.: Document expansion by query prediction. ArXiv abs/1904.08375

Then

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Huggingface’s transformers: State-of-the-art natural language processing. ArXiv abs/1910.03771
Original
2019
Later among the works it cites.
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.E.: A simple framework for contrastive learning of visual representations. ArXiv abs/2002.05709
Original
2020
Later among the works it cites.
Clark, K., Luong, M.T., Le, Q.V., Manning, C.D.: Electra: Pre-training text encoders as discriminators rather than generators. ArXiv abs/2003.10555
Original
2020
Later among the works it cites.
Dai, Z., Callan, J.P.: Context-aware document term weighting for ad-hoc search. Proceedings of The Web Conference 2020 (2020)
2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2019

Cited alongside, same era.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019), http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

2019

Cited alongside, same era.

Dai, Z., Callan, J.: Context-aware term weighting for first stage passage retrieval. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (2020)

2020

Later among the works it cites.