Potential and Perils of Large Language Models as Judges of Unstructured Textual Data

A coefficient of agreement for nominal scales

Jacob Cohen. 1960 · 1960

Earlier work this paper cites.

Answering the call for a standard reliability measure for coding data

Andrew F Hayes and Klaus Krippendorff. 2007 · 2007

Earlier work this paper cites.

Computing Krippendorff’s alpha-reliability

Klaus Krippendorff. 2011 · 2011

Earlier work this paper cites.

Interrater reliability: the kappa statistic

Mary L McHugh. 2012 · 2012

Earlier work this paper cites.

Five ways to look at Cohen’s kappa

Matthijs J Warrens. 2015 · 2015

Earlier work this paper cites.

The SAGE encyclopedia of communication research methods

Mike Allen. 2017 · 2017

Earlier work this paper cites.

Quantitative ethnography

David Williamson Shaffer. 2017 · 2017

Earlier work this paper cites.

Content analysis: An introduction to its methodology

Klaus Krippendorff. 2018 · 2018

Earlier work this paper cites.

Correlation coefficients: appropriate use and interpretation

Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018 · 2018

Earlier work this paper cites.

What to do with all those open-ended responses? Data visualization techniques for survey researchers

Jessie Rouder, Olivia Saucier, Rachel Kinder, and Matt Jans. 2021 · 2021

Earlier work this paper cites.

Data Analytics for Social Microblogging Platforms

Soumi Dutta, Asit Kumar Das, Saptarshi Ghosh, and Debabrata Samanta. 2022 · 2022

Cited alongside, same era.

Using large language models for qualitative analysis can introduce serious bias

Original

Julian Ashwin, Aditya Chhabra, and Vijayendra Rao. 2023 · 2023

Cited alongside, same era.

G-eval: Nlg evaluation using gpt-4 with better human alignment

Original

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 · 2023

Cited alongside, same era.

Aligning Large Language Models with Human Opinions through Persona Selection and Value–Belief–Norm Reasoning

Do Xuan Long, Kenji Kawaguchi, Min-Yen Kan, and Nancy F Chen. 2023 · 2023

Cited alongside, same era.

A step-by-step process of thematic analysis to develop a conceptual model in qualitative research

Muhammad Naeem, Wilson Ozuem, Kerry Howell, and Silvia Ranfagni. 2023 · 2023

Judging llm-as-a-judge with mt-bench and chatbot arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al · 2023

Later among the works it cites.

Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications

Original

Charith Chandra Sai Balne, Sreyoshi Bhaduri, Tamoghna Roy, Vinija Jain, and Aman Chadha. 2024 · 2024

Later among the works it cites.

Reconciling methodological paradigms: Employing large language models as novice qualitative research assistants in talent management research

Original

Sreyoshi Bhaduri, Satya Kapoor, Alex Gil, Anshul Mittal, and Rutu Mulkar. 2024a · 2024

Later among the works it cites.

(Multi-disciplinary) Teamwork makes the (real) dream work: Pragmatic recommendations from industry for engineering classrooms

Sreyoshi Bhaduri, Kenneth Ohnemus, Jess Blackburn, Anshul Mittal, Yan Dong, Savannah Laferriere, Robert Pulvermacher, Marina Dias, Alex Gil, Shahriar Sadighi, et al · 2024

Later among the works it cites.