2020

Overview of the TREC 2019 deep learning track

Craswell, Nick, Mitra, Bhaskar, Yilmaz, Emine et al.

Understand

The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime.

It is the first track with large human-labeled training sets, introducing two sets corresponding to two tasks, each with rigorous TREC-style blind evaluation and reusable test sets.
The document retrieval task has a corpus of 3.2 million documents with 367 thousand training queries, for which we generate a reusable test set of 43 queries.
The passage retrieval task has a corpus of 8.8 million passages with 503 thousand training queries, for which we generate a reusable test set of 43 queries.

Built on

Xlnet: Generalized autoregressive pretraining for language understanding
Original
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le · 1906
Earlier work this paper cites.
How reliable are the results of large-scale information retrieval experiments?
Justin Zobel · 1998
Earlier work this paper cites.
Umass at trec 2004: Novelty and hard
Nasreen Abdul-Jaleel, James Allan, W Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Mark D Smucker, and Courtney Wade · 2004
Earlier work this paper cites.
Indri: A language model-based search engine for complex queries
Trevor Strohman, Donald Metzler, Howard Turtle, and W Bruce Croft · 2005
Earlier work this paper cites.
High accuracy retrieval with multiple nested ranker
Irina Matveeva, Chris Burges, Timo Burkard, Andy Laucius, and Leon Wong · 2006
Earlier work this paper cites.
Bias and the limits of pooling for large collections
Chris Buckley, Darrin Dimmick, Ian Soboroff, and Ellen Voorhees · 2007
Earlier work this paper cites.
Visualizing data using t-sne
Laurens van der Maaten and Geoffrey Hinton · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Learning to rank for information retrieval
Tie-Yan Liu · 2009
Earlier work this paper cites.
The probabilistic relevance framework: Bm25 and beyond
Stephen Robertson, Hugo Zaragoza, et al · 2009
Earlier work this paper cites.
A cascade ranking model for efficient ranked retrieval
Lidan Wang, Jimmy Lin, and Donald Metzler · 2011
Earlier work this paper cites.

Similar

The arcade learning environment: An evaluation platform for general agents
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling · 2013
Cited alongside, same era.
Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton · 2015
Cited alongside, same era.
Ms marco: A human generated machine reading comprehension dataset
Original
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al · 2016
Cited alongside, same era.
Off the beaten path: Let’s replace term-based retrieval with k-nn search
Leonid Boytsov, David Novak, Yury Malkov, and Eric Nyberg · 2016
Cited alongside, same era.
Bag of tricks for efficient text classification
Original
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov · 2016
Cited alongside, same era.
Neural ranking models with weak supervision
Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W Bruce Croft · 2017
Cited alongside, same era.
Trec complex answer retrieval overview
Laura Dietz, Manisha Verma, Filip Radlinski, and Nick Craswell · 2017
Cited alongside, same era.
Learning to match using local and distributed representations of text for web search
Bhaskar Mitra, Fernando Diaz, and Nick Craswell · 2017
Cited alongside, same era.
A system for efficient high-recall retrieval
Mustafa Abualsaud, Nimesh Ghelani, Haotian Zhang, Mark D Smucker, Gordon V Cormack, and Maura R Grossman · 2018
Cited alongside, same era.
Beyond pooling
Gordon V Cormack and Maura R Grossman · 2018
Cited alongside, same era.
Critically examining the “neural hype”: Weak baselines and the additivity of effectiveness gains from neural ranking models
Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin
Cited in the paper.

Then

Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Later among the works it cites.
An introduction to neural information retrieval
Bhaskar Mitra and Nick Craswell · 2018
Later among the works it cites.
Optimizing query evaluations using reinforcement learning for web search
Corby Rosset, Damien Jose, Gargi Ghosh, Bhaskar Mitra, and Saurabh Tiwary · 2018
Later among the works it cites.
On building fair and reusable test collections using bandit techniques
Ellen M Voorhees · 2018
Later among the works it cites.
From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing
Hamed Zamani, Mostafa Dehghani, W Bruce Croft, Erik Learned-Miller, and Jaap Kamps · 2018
Later among the works it cites.
An updated duet model for passage re-ranking
Original
Bhaskar Mitra and Nick Craswell · 2019
Later among the works it cites.
Incorporating query term independence assumption for efficient retrieval and ranking using deep neural networks
Original
Bhaskar Mitra, Corby Rosset, David Hawking, Nick Craswell, Fernando Diaz, and Emine Yilmaz · 2019
Later among the works it cites.
Document expansion by query prediction
Original
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho · 2019
Later among the works it cites.
Reproducing and generalizing semantic term matching in axiomatic information retrieval
Peilin Yang and Jimmy Lin · 2019
Later among the works it cites.
Bm25 pseudo relevance feedback using anserini at waseda university
Zhaohao Zeng and Tetsuya Sakai · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…