Improving LLM-as-a-Judge Inference with the Judgment Distribution

Task complexity and contingent processing in decision making: An information search and protocol analysis

John W. Payne. 1976 · 1976

Earlier work this paper cites.

Independence of clones as a criterion for voting rules

T. N. Tideman. 1987 · 1987

Earlier work this paper cites.

The paradox of nontransitive dice

Richard P. Savage. 1994 · 1994

Earlier work this paper cites.

Statistical properties of the sample semi-variance

Shaun A. Bond and Stephen E. Satchell. 2002 · 2002

Earlier work this paper cites.

Nontransitive dice with equal means

Mark Finkelstein and Edward O. Thorp. 2006 · 2006

Earlier work this paper cites.

A new monotonic, clone-independent, reversal symmetric, and condorcet-consistent single-winner election method

Markus Schulze. 2011 · 2011

Earlier work this paper cites.

Intransitive dice

Brian Conrey, James Gabbard, Katie Grant, Andrew Liu, and Kent E. Morrison. 2013 · 2013

Earlier work this paper cites.

Intransitivity in theory and in the real world

Alexander Y Klimenko. 2015 · 2015

Earlier work this paper cites.

Training a helpful and harmless assistant with reinforcement learning from human feedback

Original

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022 · 2022

Earlier work this paper cites.

Chain of thought prompting elicits reasoning in large language models

Original

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, F. Xia, Quoc Le, and Denny Zhou. 2022 · 2022

Earlier work this paper cites.

Rankformer: Listwise learning-to-rank using listwide labels

Maarten Buyl, Paul Missault, and Pierre-Antoine Sondag. 2023 · 2023

Earlier work this paper cites.

Chateval: Towards better llm-based evaluators through multi-agent debate

Original

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shan Zhang, Jie Fu, and Zhiyuan Liu. 2023 · 2023

Earlier work this paper cites.

Batch prompting: Efficient inference with large language model apis

Zhoujun Cheng, Jungo Kasai, and Tao Yu. 2023 · 2023

Earlier work this paper cites.

A closer look into automatic evaluation using large language models

Original

Cheng-Han Chiang and Hunghuei Lee. 2023 · 2023

Earlier work this paper cites.

Ultrafeedback: Boosting language models with high-quality feedback

Original

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023 · 2023

Earlier work this paper cites.

Mistral 7b

Original

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 · 2023

Earlier work this paper cites.

Benchmarking cognitive biases in large language models as evaluators

Original

Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023 · 2023

Earlier work this paper cites.

Applying large language models and chain-of-thought for automatic scoring

Original

Gyeong-Geon Lee, Ehsan Latif, Xuansheng Wu, Ninghao Liu, and Xiaoming Zhai. 2023 · 2023

Earlier work this paper cites.

Alpacaeval: An automatic evaluator of instruction-following models

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 · 2023

Earlier work this paper cites.

Let’s verify step by step

Original

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023 · 2023

Earlier work this paper cites.

Zero-shot nlg evaluation through pairware comparisons with llms

Original

Adian Liusie, Potsawee Manakul, and Mark JF Gales. 2023 · 2023

Earlier work this paper cites.

Octopack: Instruction tuning code large language models

Original

Niklas Muennighoff, Qian Liu, Qi Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, et al. 2023 · 2023

Earlier work this paper cites.

Large language models are effective text rankers with pairwise ranking prompting

Original

Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, et al. 2023 · 2023

Earlier work this paper cites.

Xstest: A test suite for identifying exaggerated safety behaviours in large language models

Original

Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023 · 2023

Earlier work this paper cites.

Branch-solve-merge improves large language model evaluation and generation

Original

Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2023 · 2023

Cited alongside, same era.

Distributional preference learning: Understanding and accounting for hidden context in rlhf

Original

Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. 2023 · 2023

Cited alongside, same era.

Found in the middle: Permutation self-consistency improves listwise ranking in large language models

Raphael Tang, Xinyu Crystina Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. 2023 · 2023

Cited alongside, same era.

Do-not-answer: A dataset for evaluating safeguards in llms

Original

Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023 · 2023

Cited alongside, same era.

Flask: Fine-grained language model evaluation based on alignment skill sets

Wildbench: Benchmarking llms with challenging tasks from real users in the wild

Original

Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024 · 2024

Later among the works it cites.

Efficient llm comparative assessment: A product of experts framework for pairwise comparisons

Original

Adian Liusie, Vatsal Raina, Yassir Fathullah, and Mark J. F. Gales. 2024 · 2024

Later among the works it cites.

Are language model logits calibrated?

Original

Charles Lovering, Michael Krumdick, Viet Dac Lai, Nilesh Kumar, Varshini Reddy, Rik Koncel-Kedziorski, and Chris Tanner. 2024 · 2024

Later among the works it cites.

Regression aware inference with llms

Michal Lukasik, Harikrishna Narasimhan, Aditya Krishna Menon, Felix X. Yu, and Sanjiv Kumar. 2024 · 2024

Later among the works it cites.

Generative reward models