Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks

The rating of chessplayers: Past and present

Arpad E Elo and Sam Sloan. 1978 · 1978

Earlier work this paper cites.

Influences on clinical judgement in mental health nursing

Peter J Martin. 1999 · 1999

Earlier work this paper cites.

BLEU: a Method for Automatic Evaluation of Machine Translation. 311–318

Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002 · 2002

Earlier work this paper cites.

ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out . Association for Computational Linguistics, Barcelona, Spain, 74–81

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

Design and evaluation guidelines for mental health technologies

Gavin Doherty, David Coyle, and Mark Matthews. 2010 · 2010

Earlier work this paper cites.

Deep reinforcement learning from human preferences

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017 · 2017

Earlier work this paper cites.

Reflecting on reflexive thematic analysis

Virginia Braun and Victoria Clarke. 2019 · 2019

Earlier work this paper cites.

Clinical Judgement: an investigation of clinical decision-making

Benjamin Paul Michael. 2019 · 2019

Earlier work this paper cites.

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Association for Computational Linguistics, Minneapolis, Minnesota, 4149–4158

Original

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019 · 2019

Earlier work this paper cites.

BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020 · 2020

Earlier work this paper cites.

On the opportunities and risks of foundation models

Original

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al · 2021

Earlier work this paper cites.

Precision nutrition: A systematic literature review

Daniel Kirk, Cagatay Catal, and Bedir Tekinerdogan. 2021 · 2021

Earlier work this paper cites.

Journal of Human Nutrition and Dietetics 34, 1 (2021), 124–133

Ruth Vo, M Smith, and N Patton. 2021 · 2021

Earlier work this paper cites.

Bridging the Gap between UX Practitioners’ work practices and AI-enabled design support tools. In CHI Conference on Human Factors in Computing Systems Extended Abstracts . 1–7

Yuwen Lu, Chengzhi Zhang, Iris Zhang, and Toby Jia-Jun Li. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023

Earlier work this paper cites.

ChainForge: An open-source visual programming environment for prompt engineering. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology . 1–3

Ian Arawjo, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023 · 2023

Earlier work this paper cites.

The now and future of ChatGPT and GPT in psychiatry

Szu-Wei Cheng, Chung-Wen Chang, Wan-Jung Chang, Hao-Wei Wang, Chih-Sung Liang, Taishiro Kishimoto, Jane Pei-Chen Chang, John S Kuo, and Kuan-Pin Su. 2023 · 2023

Earlier work this paper cites.

ChatGPT as a virtual dietitian: Exploring its potential as a tool for improving nutrition knowledge

Manuel B Garcia. 2023 · 2023

Earlier work this paper cites.

Patat: Human-ai collaborative qualitative coding with explainable interactive rule synthesis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–19

Simret Araya Gebreegziabher, Zheng Zhang, Xiaohang Tang, Yihao Meng, Elena L Glassman, and Toby Jia-Jun Li. 2023 · 2023

Earlier work this paper cites.

Rethinking large language models in mental health applications

Original

Shaoxiong Ji, Tianlin Zhang, Kailai Yang, Sophia Ananiadou, and Erik Cambria. 2023 · 2023

Earlier work this paper cites.

Evallm: Interactive evaluation of large language model prompts on user-defined criteria

Original

Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2023 · 2023

Earlier work this paper cites.

An introduction to generative artificial intelligence in mental health care: considerations and guidance

Darlene R King, Guransh Nanda, Joel Stoddard, Allison Dempsey, Sarah Hergert, Jay H Shore, and John Torous. 2023 · 2023

Cited alongside, same era.

Comparison of answers between ChatGPT and human dieticians to common nutrition questions

Daniel Kirk, Elise van Eijnatten, and Guido Camps. 2023 · 2023

Cited alongside, same era.

Chatharuhi: Reviving anime character in reality via large language model

Original

Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, et al · 2023

Cited alongside, same era.

Chatcounselor: A large language models for mental health support

Original

June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. 2023 · 2023

Cited alongside, same era.

The credibility of dietary advice formulated by ChatGPT: robo-diets for people with food allergies

Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning

Original

Simret Araya Gebreegziabher, Kuangshi Ai, Zheng Zhang, Elena L Glassman, and Toby Jia-Jun Li. 2024a · 2024

Closest in time.

Supporting Co-Adaptive Machine Teaching through Human Concept Learning and Cognitive Theories

Original

Simret Araya Gebreegziabher, Yukun Yang, Elena L Glassman, and Toby Jia-Jun Li. 2024b · 2024

Closest in time.

Overview of Gemini: Large Multimodal Models

Google Research. 2024 · 2024

Closest in time.

Large language models in mental health care: a scoping review

Original

Yining Hua, Fenglin Liu, Kailai Yang, Zehan Li, Yi-han Sheu, Peilin Zhou, Lauren V Moran, Sophia Ananiadou, and Andrew Beam. 2024 · 2024

Closest in time.

" I’m categorizing LLM as a productivity tool": Examining ethics of LLM use in HCI research practices