Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences

ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons

Original

Margaret Li, Jason Weston, and Stephen Roller. 2019 · 1909

Earlier work this paper cites.

NASA-task load index (NASA-TLX); 20 years later

Sandra G Hart. 2006 · 2006

Earlier work this paper cites.

A general inductive approach for analyzing qualitative evaluation data

David R Thomas. 2006 · 2006

Earlier work this paper cites.

Accounting for diversity in subjective judgments

Evangelos Karapanos, Jean-Bernard Martens, and Marc Hassenzahl. 2009 · 2009

Earlier work this paper cites.

Gestalt: integrated support for implementation and analysis in machine learning

Kayur Patel, Naomi Bancroft, Steven M Drucker, James Fogarty, Amy J Ko, and James Landay. 2010 · 2010

Earlier work this paper cites.

Bigbench: Towards an industry standard benchmark for big data analytics

Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Arno Jacobsen. 2013 · 2013

Earlier work this paper cites.

The future of crowd work

Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013 · 2013

Earlier work this paper cites.

Ice: enabling non-experts to build models interactively for large-scale lopsided problems

Patrice Simard, David Chickering, Aparna Lakshmiratan, Denis Charles, Léon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, and Jina Suh. 2014 · 2014

Earlier work this paper cites.

Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems

Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. 2020 · 2020

Earlier work this paper cites.

Design justice: Community-led practices to build the worlds we need

Sasha Costanza-Chock. 2020 · 2020

Earlier work this paper cites.

GPT-3: Its nature, scope, limits, and consequences

Luciano Floridi and Massimo Chiriatti. 2020 · 2020

Earlier work this paper cites.

Measuring massive multitask language understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2020

Earlier work this paper cites.

Evaluating the Factual Consistency of Abstractive Text Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 9332–9346

Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020 · 2020

Earlier work this paper cites.

Retrieval-augmented generation for knowledge-intensive nlp tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al · 2020

Earlier work this paper cites.

Summeval: Re-evaluating summarization evaluation

Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021 · 2021

Earlier work this paper cites.

Manipulating and measuring model interpretability

Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach. 2021 · 2021

Earlier work this paper cites.

Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022 · 2022

Earlier work this paper cites.

AI Assisted Data Labeling with Interactive Auto Label

Michael Desmond, Michelle Brachman, Evelyn Duesterwald, Casey Dugan, Narendra Nath Joshi, Qian Pan, and Carolina Spina. 2022 · 2022

Earlier work this paper cites.

Holistic evaluation of language models

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al · 2022

Earlier work this paper cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022

Cited alongside, same era.

A Closer Look into Using Large Language Models for Automatic Evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 8928–8942

Cheng-Han Chiang and Hung-yi Lee. 2023 · 2023

Cited alongside, same era.

Can Large Language Models Be an Alternative to Human Evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, Toronto, Canada, 15607–15631

Cheng-Han Chiang and Hung yi Lee. 2023 · 2023

Cited alongside, same era.

Retrieval-augmented generation for large language models: A survey

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023 · 2023

Cited alongside, same era.

Alpacafarm: A simulation framework for methods that learn from human feedback

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2024 · 2024

Closest in time.

HELM: Holistic Evaluation of Language Models

Center for Research on Foundation Models. 2024 · 2024

Closest in time.

Incentives, Framing, and Reliance on Algorithmic Advice: An Experimental Study

Ben Greiner, Philipp Grünwald, Thomas Lindner, Georg Lintner, and Martin Wiernsperger. 2024 · 2024

Closest in time.

An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers

Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. 2024 · 2024

Closest in time.

Llm comparator: Visual analytics for side-by-side evaluation of large language models

Minsuk Kahng, Ian Tenney, Mahima Pushkarna, Michael Xieyang Liu, James Wexler, Emily Reif, Krystal Kallarackal, Minsuk Chang, Michael Terry, and Lucas Dixon. 2024 · 2024

Closest in time.