Can We Use Large Language Models to Fill Relevance Judgment Holes?

The Cranfield Tests on Index Language Devices

Cyril W. Cleverdon. 1967 · 1967

Earlier work this paper cites.

Report on the Need for and Provision of an ’Ideal’ Information Retrieval Test Collection

Karen Sparck-Jones and C. J. Van Rijsbergen. 1975 · 1975

Earlier work this paper cites.

Efficient construction of large test collections. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Melbourne, Australia) (SIGIR ’98) . Association for Computing Machinery, New York, NY, USA, 282–289

Gordon V. Cormack, Christopher R. Palmer, and Charles L. A. Clarke. 1998 · 1998

Earlier work this paper cites.

Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Sheffield, United Kingdom) (SIGIR ’04) . 25–32

Chris Buckley and Ellen M. Voorhees. 2004 · 2004

Earlier work this paper cites.

Information retrieval system evaluation: effort, sensitivity, and reliability. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Salvador, Brazil) (SIGIR ’05) . 162–169

Mark Sanderson and Justin Zobel. 2005 · 2005

Earlier work this paper cites.

TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing)

Ellen M. Voorhees and Donna K. Harman. 2005 · 2005

Earlier work this paper cites.

Minimal test collections for retrieval evaluation. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Seattle, Washington, USA) (SIGIR ’06) . Association for Computing Machinery, New York, NY, USA, 268–275

Ben Carterette, James Allan, and Ramesh Sitaraman. 2006 · 2006

Earlier work this paper cites.

Inferring document relevance from incomplete information. In Proceedings of the Sixteenth ACM Conference on Conference on Information and Knowledge Management (Lisbon, Portugal) (CIKM ’07) . Association for Computing Machinery, New York, NY, USA, 633–642

Javed A. Aslam and Emine Yilmaz. 2007 · 2007

Earlier work this paper cites.

A Retrieval Evaluation Methodology for Incomplete Relevance Assessments. In Advances in Information Retrieval, 29th European Conference on IR Research, ECIR 2007 (Rome, Italy) (Lecture Notes in Computer Science, Vol. 4425) . Springer, 271–282

Mark Baillie, Leif Azzopardi, and Ian Ruthven. 2007 · 2007

Earlier work this paper cites.

Strategic system comparisons via targeted relevance judgments. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Amsterdam, The Netherlands) (SIGIR ’07) . 375–382

Alistair Moffat, William Webber, and Justin Zobel. 2007 · 2007

Earlier work this paper cites.

Evaluating epistemic uncertainty under incomplete assessments

Mark Baillie, Leif Azzopardi, and Ian Ruthven. 2008 · 2008

Cited alongside, same era.

MS MARCO: A Human Generated Machine Reading Comprehension Dataset. In NIPS

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Xiaodong Liu Jianfeng Gao, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016 · 2016

Cited alongside, same era.

The effect of pooling and evaluation depth on IR metrics

Xiaolu Lu, Alistair Moffat, and J Shane Culpepper. 2016 · 2016

Cited alongside, same era.

Fixed-Cost Pooling Strategies Based on IR Evaluation Measures. In Advances in Information Retrieval . 357–368

Aldo Lipani, Joao Palotti, Mihai Lupu, Florina Piroi, Guido Zuccon, and Allan Hanbury. 2017 · 2017

Cited alongside, same era.

A Theoretical Framework for Conversational Search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (Oslo, Norway) (CHIIR ’17) . Association for Computing Machinery, New York, NY, USA, 117–126

One-shot labeling for automatic relevance estimation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2230–2235

Sean MacAvaney and Luca Soldaini. 2023 · 2023

Later among the works it cites.

RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!

Original

Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023 · 2023

Later among the works it cites.

Large language models can accurately predict searcher preferences

Original

Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023 · 2023

Later among the works it cites.

LLaMA: Open and Efficient Foundation Language Models

Original

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al · 2023

Later among the works it cites.