Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation

A coefficient of agreement for nominal scales

Jacob Cohen. 1960 · 1960

Earlier work this paper cites.

Hedges: A study in meaning criteria and the logic of fuzzy concepts

George Lakoff. 1973 · 1973

Earlier work this paper cites.

An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers

J Richard Landis and Gary G Koch. 1977 · 1977

Earlier work this paper cites.

The many meanings of uncertainty in illness: Toward a systematic accounting

Austin S Babrow, Chris R Kasch, and Leigh A Ford. 1998 · 1998

Earlier work this paper cites.

Communication in the management of uncertainty: The case of persons living with hiv or aids

Dale E Brashers, Judith L Neidig, Stephen M Haas, Linda K Dobbs, Linda W Cardillo, and Jane A Russell. 2000 · 2000

Earlier work this paper cites.

Stance and engagement: A model of interaction in academic discourse

Ken Hyland. 2005 · 2005

Earlier work this paper cites.

Interrater reliability: the kappa statistic

Mary L McHugh. 2012 · 2012

Earlier work this paper cites.

Disciplinary discourses: Writer stance in research articles

Ken Hyland. 2014 · 2014

Earlier work this paper cites.

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017 · 2017

Earlier work this paper cites.

Natural questions: a benchmark for question answering research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019 · 2019

Earlier work this paper cites.

Explanation in artificial intelligence: Insights from the social sciences

Tim Miller. 2019 · 2019

Earlier work this paper cites.

A general language assistant as a laboratory for alignment

Original

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021 · 2021

Earlier work this paper cites.

Truthful ai: Developing and governing ai that does not lie

Original

Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021 · 2021

Earlier work this paper cites.

Language models (mostly) know what they know

Original

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022 · 2022

Earlier work this paper cites.

Teaching models to express their uncertainty in words

Original

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022 · 2022

Earlier work this paper cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Earlier work this paper cites.

Missing information, unresponsive authors, experimental flaws: The impossibility of assessing the reproducibility of previous human evaluations in nlp

Original

Anya Belz, Craig Thomson, Ehud Reiter, Gavin Abercrombie, Jose M Alonso-Moral, Mohammad Arvan, Anouck Braggaar, Mark Cieliebak, Elizabeth Clark, Kees van Deemter, et al. 2023 · 2023

Earlier work this paper cites.

Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images

Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. 2023 · 2023

Earlier work this paper cites.

Can large language models be an alternative to human evaluations?

Original

Cheng-Han Chiang and Hung-yi Lee. 2023 · 2023

Cited alongside, same era.

A diachronic perspective on user trust in ai under uncertainty

Shehzaad Dhuliawala, Vilém Zouhar, Mennatallah El-Assady, and Mrinmaya Sachan. 2023 · 2023

Cited alongside, same era.

Shifting attention to relevance: Towards the uncertainty estimation of large language models

Original

Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2023 · 2023

Cited alongside, same era.

Llm-blender: Ensembling large language models with pairwise ranking and generative fusion

Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023 · 2023

Cited alongside, same era.

Challenges and applications of large language models

Original

Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023 · 2023

Judgelm: Fine-tuned large language models are scalable judges

Original

Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023 · 2023

Later among the works it cites.

Mars: Meaning-aware response scoring for uncertainty estimation in generative llms

Original

Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. 2024 · 2024

Closest in time.

A survey on evaluation of large language models

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024 · 2024

Closest in time.

The llama 3 herd of models

Original

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024 · 2024

Closest in time.

Alpacafarm: A simulation framework for methods that learn from human feedback