Fetching the paper…

Reading the bibliography…

2024

LLMJudge: LLMs for Relevance Judgments

Rahmani, Hossein A., Yilmaz, Emine, Craswell, Nick et al.

Original
PDF
Post on X

Understand

The LLMJudge challenge is organized as part of the LLM4Eval workshop at SIGIR 2024.

Test collections are essential for evaluating information retrieval (IR) systems.
The evaluation and tuning of a search system is largely based on relevance labels, which indicate whether a document is useful for a specific search and user.
However, collecting relevance judgments on a large scale is costly and resource-intensive.

LLMJudge: LLMs for Relevance Judgments · Around

Built on

Perspectives on large language models for relevance judgment,
G. Faggioli, L. Dietz, C. L. Clarke, G. Demartini, M. Hagen, C. Hauff, N. Kando, E. Kanoulas, M. Potthast, B. Stein, et al., · 2023
Earlier work this paper cites.
Large language models can accurately predict searcher preferences,
Original
P. Thomas, S. Spielman, N. Craswell, B. Mitra, · 2023
Earlier work this paper cites.

Similar

Overview of the trec 2023 deep learning track,
N. Craswell, B. Mitra, E. Yilmaz, H. A. Rahmani, D. Campos, J. Lin, E. M. Voorhees, I. Soboroff, · 2023
Cited alongside, same era.
Llm4eval: Large language model for evaluation in ir,
H. A. Rahmani, C. Siro, M. Aliannejadi, N. Craswell, C. L. A. Clarke, G. Faggioli, B. Mitra, P. Thomas, E. Yilmaz, · 2024
Cited alongside, same era.

Then

Synthetic test collections for retrieval evaluation,
Original
H. A. Rahmani, N. Craswell, E. Yilmaz, B. Mitra, D. Campos, · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…