Fetching the paper…
Reading the bibliography…
The fundamental property of Cranfield-style evaluations, that system rankings are stable even when assessors disagree on individual relevance decisions, was validated on traditional test collections.
A Note on the Pseudo-Mathematics of Relevance
Mortimer Taube. 1965 · 1965
Earlier work this paper cites.
Factors Determining the Performance of Indexing Systems. Volume I. Design. Part 2. Appendices
C. Cleverdon, J. Mills, and M. Keen. 1966 · 1966
Earlier work this paper cites.
Opening the Black Box of Relevance
Carlos A. Cuadra and Robert V. Katter. 1967 · 1967
Earlier work this paper cites.
A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching. Final Report to the National Science Foundation. Volume I
Alan M. Rees and Douglas G. Schultz. 1967 · 1967
Earlier work this paper cites.
Relevance assessments and retrieval system evaluation
M.E. Lesk and G. Salton. 1968 · 1968
Earlier work this paper cites.
Performance measures for information retrieval systems—an experimental approach
John J. Regazzi. 1988 · 1988
Earlier work this paper cites.
A Re-Examination of Relevance: Toward a Dynamic, Situational Definition ∗ \ast
Linda Schamber, Michael B. Eisenberg, and Michael S. Nilan. 1990 · 1990
Earlier work this paper cites.
Variations in relevance judgments and the evaluation of retrieval performance
Robert Burgin. 1992 · 1992
Earlier work this paper cites.
Overview of the Fourth Text REtrieval Conference (TREC-4). In Proceedings of The Fourth Text REtrieval Conference, TREC 1995, Gaithersburg, Maryland, USA, November 1-3, 1995 (NIST Special Publication, Vol. 500-236) , Donna K. Harman (Ed.). National Institute of Standards and Technology (NIST)
Donna Harman. 1995 · 1995
Earlier work this paper cites.
Okapi at TREC-4. In Proceedings of The Fourth Text REtrieval Conference, TREC 1995, Gaithersburg, Maryland, USA, November 1-3, 1995 (NIST Special Publication, Vol. 500-236) , Donna K. Harman (Ed.). National Institute of Standards and Technology (NIST)
Stephen E. Robertson, Steve Walker, Micheline Hancock-Beaulieu, Mike Gatford, and A. Payne. 1995 · 1995
Earlier work this paper cites.
Relevance: The Whole History
Stefano Mizzaro. 1997 · 1997
Earlier work this paper cites.
Overview of the Sixth Text REtrieval Conference (TREC-6)
Ellen M. Voorhees and Donna K. Harman. 1997 · 1997
Earlier work this paper cites.
Improvements that don’t add up: ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (Hong Kong, China) (CIKM ’09) . Association for Computing Machinery, New York, NY, USA, 601–610
Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009 · 1998
Earlier work this paper cites.
Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR ’98) . Association for Computing Machinery, New York, NY, USA, 315–323
Ellen M. Voorhees. 1998 · 1998
Earlier work this paper cites.
How reliable are the results of large-scale information retrieval experiments?. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Melbourne, Australia) (SIGIR ’98) . Association for Computing Machinery, New York, NY, USA, 307–314
Justin Zobel. 1998 · 1998
Earlier work this paper cites.
Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness
Ellen Voorhees. 2000 · 2000
Earlier work this paper cites.
Cumulated gain-based evaluation of IR techniques
Kalervo Järvelin and Jaana Kekäläinen. 2002 · 2002
Earlier work this paper cites.
Comparing metrics across TREC and NTCIR: the robustness to system bias. In Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM 2008, Napa Valley, California, USA, October 26-30, 2008 , James G. Shanahan, Sihem Amer-Yahia, Ioana Manolescu, Yi Zhang, David A. Evans, Aleksander Kolcz, Key-Sun Choi, and Abdur Chowdhury (Eds.). ACM, 581–590
Tetsuya Sakai. 2008 · 2008
Earlier work this paper cites.
Expected reciprocal rank for graded relevance. In CIKM ’09: Proceeding of the 18th ACM conference on Information and knowledge management . New York, NY, USA, 621–630
Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. 2009 · 2009
Cited alongside, same era.
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020 · 2010
Cited alongside, same era.
TREC-Style Evaluations. In Information Retrieval Meets Information Visualization - PROMISE Winter School 2012, Zinal, Switzerland, January 23-27, 2012, Revised Tutorial Lectures (Lecture Notes in Computer Science, Vol. 7757) , Maristella Agosti, Nicola Ferro, Pamela Forner, Henning Müller, and Giuseppe Santucci (Eds.). Springer, 97–115
Donna Harman. 2012 · 2012
Cited alongside, same era.
ChatNoir: a search engine for the ClueWeb09 corpus. In The 35th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR ’12, Portland, OR, USA, August 12-16, 2012 , William R. Hersh, Jamie Callan, Yoelle Maarek, and Mark Sanderson (Eds.). ACM, 1004
University of Amsterdam at TREC 2021: Deep Learning Track
Jaap Kamps and David Rau. [n. d.] · 2021
Later among the works it cites.
From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (Madrid, Spain) (SIGIR ’22) . Association for Computing Machinery, New York, NY, USA, 2353–2359
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2022 · 2022
Later among the works it cites.
ClueWeb22: 10 Billion Web Documents with Rich Information. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022 , Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (Eds.). ACM, 3360–3362
Arnold Overwijk, Chenyan Xiong, and Jamie Callan. 2022 · 2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Martin Potthast, Matthias Hagen, Benno Stein, Jan Graßegger, Maximilian Michel, Martin Tippmann, and Clement Welsch. 2012 · 2012
Cited alongside, same era.
Distilling the Knowledge in a Neural Network
Geoffrey Hinton. 2015 · 2015
Cited alongside, same era.
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016 (CEUR Workshop Proceedings, Vol. 1773) , Tarek Richard Besold, Antoine Bordes, Artur S. d’Avila Garcez, and Greg Wayne (Eds.). CEUR-WS.org
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016 · 2016
Cited alongside, same era.
Netflix recommendations: Beyond the 5 stars (part 1)
Xavier Amatriain and Justin Basilico. 2017 · 2017
Cited alongside, same era.
A System for Efficient High-Recall Retrieval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval . ACM, 1317–1320
Mustafa Abualsaud, Nimesh Ghelani, Haotian Zhang, Mark D Smucker, Gordon V Cormack, and Maura R Grossman. 2018 · 2018
Cited alongside, same era.
Overview of the TREC 2019 Deep Learning Track. In 28th International Text Retrieval Conference, TREC 2019, Gaithersburg, Maryland, USA (NIST Special Publication) , Ellen M. Voorhees and Angela Ellis (Eds.). National Institute of Standards and Technology (NIST)
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2019 · 2019
Cited alongside, same era.
How to Run an Evaluation Task - With a Primary Focus on Ad Hoc Information Retrieval
Tetsuya Sakai. 2019 · 2019
Cited alongside, same era.
The Evolution of Cranfield
Ellen M. Voorhees. 2019 · 2019
Cited alongside, same era.
Overview of the TREC 2020 Deep Learning Track. In Proceedings of the 29th Text REtrieval Conference, TREC 2020, Virtual Event, Gaithersburg, MD, USA, November 16-20, 2020 (NIST Special Publication, Vol. 1266) , Ellen M. Voorhees and Angela Ellis (Eds.). National Institute of Standards and Technology (NIST)
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2020 · 2020
Cited alongside, same era.
Ronak Pradeep, Yuqi Liu, Xinyu Zhang, Yilin Li, Andrew Yates, and Jimmy Lin. 2022 · 2022
Later among the works it cites.
LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiaolong Huang, Binxing Jiao, Linjun Yang, and Daxin Jiang. 2022 · 2022
Later among the works it cites.
Too Many Relevants: Whither Cranfield Test Collections?. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (Madrid, Spain) (SIGIR ’22) . Association for Computing Machinery, New York, NY, USA, 2970–2980
Ellen M. Voorhees, Nick Craswell, and Jimmy Lin. 2022 · 2022
Later among the works it cites.
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval. In Annual Meeting of the Association for Computational Linguistics
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022 · 2022
Later among the works it cites.
Bootstrapped nDCG Estimation in the Presence of Unjudged Documents. In Advances in Information Retrieval - 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 13980) , Jaap Kamps, Lorraine Goeuriot, Fabio Crestani, Maria Maistro, Hideo Joho, Brian Davis, Cathal Gurrin, Udo Kruschwitz, and Annalina Caputo (Eds.). Springer, 313–329
Maik Fröbe, Lukas Gienapp, Martin Potthast, and Matthias Hagen. 2023 · 2023
Later among the works it cites.
RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!
Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023 · 2023
Later among the works it cites.
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 14918–14937
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023 · 2023
Later among the works it cites.
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, Toronto, Canada, 2244–2258
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2023 · 2023
Later among the works it cites.
Contextual masked auto-encoder for dense passage retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37. 4738–4746
Xing Wu, Guangyuan Ma, Meng Lin, Zijia Lin, Zhongyuan Wang, and Songlin Hu. 2023 · 2023
Later among the works it cites.
Report on the Collab-a-Thon at ECIR 2024
Sean MacAvaney, Adam Roegiest, Aldo Lipani, Andrew Parry, Björn Engelmann, Christin Katharina Kreutz, Chuan Meng, Erlend Frayling, Eugene Yang, Ferdinand Schlatt, Guglielmo Faggioli, Harrisen Scells, Iana Atanassova, Jana Friese, Janek Bevendorff, Javier Sanz-Cruzado, Johanne Trippas, Kanaad Pathak, Kaustubh D. Dhole, Leif Azzopardi, Maik Fröbe, Marc Bertin, Nishchal Prasad, Saber Zerhoudi, Shuai Wang, Shubham Chatterjee, Thomas Jänich, Udo Kruschwitz, Xi Wang, and Zijun Long. 2024 · 2024
Later among the works it cites.
Top-Down Partitioning for Efficient List-Wise Ranking
Andrew Parry, Sean MacAvaney, and Debasis Ganguly. 2024 · 2024
Later among the works it cites.
Ferdinand Schlatt, Maik Fröbe, Harrisen Scells, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Benno Stein, Martin Potthast, and Matthias Hagen. 2024a · 2024
Later among the works it cites.
Don’t Use LLMs to Make Relevance Judgments
Ian Soboroff. 2024 · 2024
Later among the works it cites.
Listwise Generative Retrieval Models via a Sequential Learning Process
Yubao Tang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, and Xueqi Cheng. 2024 · 2024
Later among the works it cites.