Human-Centered Design Recommendations for LLM-as-a-Judge

Advancing the rationality debate

Keith E. Stanovich and Richard F. West. 2000 · 2000

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

The what-if tool: Interactive probing of machine learning models

James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Viégas, and Jimbo Wilson. 2019 · 2019

Earlier work this paper cites.

Errudite: Scalable, reproducible, and testable error analysis

Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. 2019 · 2019

Earlier work this paper cites.

Semi-automated data labeling

Michael Desmond, Evelyn Duesterwald, Kristina Brimijoin, Michelle Brachman, and Qian Pan. 2021 · 2020

Earlier work this paper cites.

On training instance selection for few-shot neural text generation

Original

Ernie Chang, Xiaoyu Shen, Hui-Syuan Yeh, and Vera Demberg. 2021 · 2021

Earlier work this paper cites.

Results of wmt22 metrics shared task: Stop using bleu–neural metrics are better and more robust

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André FT Martins. 2022 · 2022

Cited alongside, same era.

Holistic evaluation of language models

Original

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022 · 2022

Cited alongside, same era.

Selective annotation makes language models better few-shot learners

Original

Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2022 · 2022

Cited alongside, same era.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 · 2022

Cited alongside, same era.

Alpacaeval: An automatic evaluator of instruction-following models

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023 · 2023

Later among the works it cites.

Constitutionmaker: Interactively critiquing large language models by converting feedback into principles

Original

Savvas Petridis, Ben Wedin, James Wexler, Aaron Donsbach, Mahima Pushkarna, Nitesh Goyal, Carrie J Cai, and Michael Terry. 2023 · 2023

Later among the works it cites.

Interface design for crowdsourcing hierarchical multi-label text annotations

Rickard Stureborg, Bhuwan Dhingra, and Jun Yang. 2023 · 2023

Later among the works it cites.

Evaluating nlg evaluation metrics: A measurement theory perspective

Original

Ziang Xiao, Susu Zhang, Vivian Lai, and Q Vera Liao. 2023 · 2023

Later among the works it cites.

Judging llm-as-a-judge with mt-bench and chatbot arena

Original