2020

Pre-training Tasks for Embedding-based Large-scale Retrieval

Chang, Wei-Cheng, Yu, Felix X., Chang, Yin-Wen et al.

Understand

We consider the large-scale query-document retrieval problem: given a query (e.g., a question), return the set of relevant documents (e.g., paragraphs containing the answer) from a large document corpus.

This problem is often solved in two steps.
The retrieval phase first reduces the solution space, returning a subset of candidate documents.
The scoring phase then re-ranks the documents.

Built on

Adaptive importance sampling to accelerate training of a neural probabilistic language model
Yoshua Bengio and Jean-Sébastien Senécal · 2008
Earlier work this paper cites.
The probabilistic relevance framework: BM25 and beyond
Stephen Robertson, Hugo Zaragoza, et al · 2009
Earlier work this paper cites.
Yahoo! learning to rank challenge overview
Olivier Chapelle and Yi Chang · 2011
Earlier work this paper cites.
Distributed representations of sentences and documents
Quoc Le and Tomas Mikolov · 2014
Earlier work this paper cites.
Asymmetric LSH (ALSH) for sublinear time maximum inner product search (mips)
Anshumali Shrivastava and Ping Li · 2014
Earlier work this paper cites.
A large annotated corpus for learning natural language inference
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning · 2015
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Earlier work this paper cites.
Skip-thought vectors
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler · 2015
Earlier work this paper cites.
Strategies for training large vocabulary neural language models
Welin Chen, David Grangier, and Michael Auli · 2016
Earlier work this paper cites.
Together we stand: Siamese networks for similar question retrieval
Arpita Das, Harish Yenala, Manoj Chinnakotla, and Manish Shrivastava · 2016
Earlier work this paper cites.
Quantization based fast inner product search
Ruiqi Guo, Sanjiv Kumar, Krzysztof Choromanski, and David Simcha · 2016
Earlier work this paper cites.
MS MARCO: A human-generated machine reading comprehension dataset
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng · 2016
Earlier work this paper cites.
Squad: 100,000+ questions for machine comprehension of text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang · 2016
Earlier work this paper cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al · 2016
Earlier work this paper cites.

Similar

Reading Wikipedia to answer open-domain questions
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes · 2017
Cited alongside, same era.
Efficient softmax approximation for gpus
Edouard Grave, Armand Joulin, Moustapha Cissé, Hervé Jégou, et al · 2017
Cited alongside, same era.
Efficient natural language response suggestion for smart reply
Original
Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil · 2017
Cited alongside, same era.
Discourse-based objectives for fast unsupervised sentence representation learning
Original
Yacine Jernite, Samuel R Bowman, and David Sontag · 2017
Cited alongside, same era.
Few-shot learning through an information retrieval lens
Eleni Triantafillou, Richard Zemel, and Raquel Urtasun · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Anserini: Enabling the use of lucene for information retrieval research
Peilin Yang, Hui Fang, and Jimmy Lin · 2017
Cited alongside, same era.
Selection of negative samples for one-class matrix factorization
Hsiang-Fu Yu, Mikhail Bilenko, and Chih-Jen Lin · 2017
Cited alongside, same era.
Adaptive sampled softmax with kernel based sampling
Guy Blanc and Steffen Rendle · 2018
Cited alongside, same era.
Universal sentence encoder
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al · 2018
Cited alongside, same era.
An efficient framework for learning sentence representations
Lajanugen Logeswaran and Honglak Lee · 2018
Cited alongside, same era.
Training millions of personalized dialogue agents
Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes · 2018
Cited alongside, same era.

Then

X-BERT: eXtreme multi-label text with BERT
Original
Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, and Inderjit Dhillon · 2019
Later among the works it cites.
Multi-step retriever-reader interaction for scalable open-domain question answering
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum · 2019
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Later among the works it cites.
Retrieve, read, rerank: Towards end-to-end multi-document reading comprehension
Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li · 2019
Later among the works it cites.
Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring
Original
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston · 2019
Later among the works it cites.
Slice: Scalable linear extreme classifiers trained on 100 million labels for related searches
Himanshu Jain, Venkatesh Balasubramanian, Bhanu Chunduri, and Manik Varma · 2019
Later among the works it cites.
Efficient training on very large corpora via gramian estimation
Walid Krichene, Nicolas Mayoraz, Steffen Rendle, Li Zhang, Xinyang Yi, Lichan Hong, Ed Chi, and John Anderson · 2019
Later among the works it cites.
Natural questions: a benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al · 2019
Later among the works it cites.
Latent retrieval for weakly supervised open domain question answering
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova · 2019
Later among the works it cites.
RoBERTa: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Later among the works it cites.
Stochastic negative mining for learning with large output spaces
Sashank J Reddi, Satyen Kale, Felix Yu, Dan Holtmann-Rice, Jiecao Chen, and Sanjiv Kumar · 2019
Later among the works it cites.
Sentence-BERT: Sentence embeddings using siamese BERT-networks
Nils Reimers and Iryna Gurevych · 2019
Later among the works it cites.
Distilling task-specific knowledge from BERT into simple neural networks
Original
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin · 2019
Later among the works it cites.
End-to-end open-domain question answering with BERTserini
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…