Fetching the paper…

Reading the bibliography…

2025

The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge

Marioriyad, Arash, Rohban, Mohammad Hossein, Baghshah, Mahdieh Soleymani

Original
PDF
Post on X

Understand

Large language models (LLMs) are increasingly deployed as automatic judges to evaluate system outputs in tasks such as summarization, dialogue, and creative writing.

A faithful judge should base its verdicts solely on response quality and explicitly acknowledge the factors shaping its decision.
We show that current LLM judges fail on both counts by relying on shortcuts introduced in the prompt.
Our study uses two evaluation datasets: ELI5, a benchmark for long-form question answering, and LitBench, a recent benchmark for creative writing.

The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge · Around

Built on

ELI5: Long form question answering
A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli · 2019
Earlier work this paper cites.
Summeval: Re-evaluating summarization evaluation
A. R. Fabbri, W. Krysciński, B. McCann, C. Xiong, R. Socher, and D. Radev · 2021
Earlier work this paper cites.
Human-bot comparison as an evaluation framework for dialogue
S. Mehri and M. Eskenazi · 2022
Earlier work this paper cites.
G-eval: General evaluation of language models
X. Liu et al · 2023
Earlier work this paper cites.
Verbosity bias in preference labeling by large language models
Original
K. Saito, A. Wachi, K. Wataoka, and Y. Akimoto · 2023
Earlier work this paper cites.

Similar

Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting
Original
M. Turpin, J. Michael, E. Perez, and S. R. Bowman · 2023
Cited alongside, same era.
Chatbot arena: A human-ai comparative judgment platform
L. Zheng et al · 2023
Cited alongside, same era.
Length-controlled alpacaeval: A simple way to debias automatic evaluators
Original
Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto · 2024
Cited alongside, same era.
Llm evaluators recognize and favor their own generations
Original
A. Panickssery, S. R. Bowman, and S. Feng · 2024
Cited alongside, same era.
Judging the judges: A systematic study of position bias in llm-as-a-judge
Original
L. Shi, C. Ma, W. Liang, W. Ma, and S. Vosoughi · 2024
Cited alongside, same era.
Judging LLM/as-a-judge with MT-bench and chatbot arena
Original
L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica
Cited in the paper.

Then

Self-preference bias in llm-as-a-judge
Original
K. Wataoka, T. Takahashi, and R. Ri · 2024
Later among the works it cites.
Chain-of-thought reasoning in the wild is not always faithful
Original
I. Arcuschin, J. Janiak, R. Krzyzanowski, S. Rajamanoharan, N. Nanda, and A. Conmy · 2025
Closest in time.
Reasoning models don’t always say what they think
Original
Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, A. Somani, P. Hase, M. Wagner, F. Roger, et al · 2025
Closest in time.
LitBench: A benchmark and dataset for reliable evaluation of creative writing
Original
D. Fein, S. Russo, V. Xiang, K. Jolly, R. Rafailov, and N. Haber · 2025
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…