Can Unconfident LLM Annotations Be Used for Confident Conclusions?

Mixed-initiative interaction

James E Allen, Curry I Guinn, and Eric Horvitz. 1999 · 1999

Earlier work this paper cites.

Asymptotic statistics , volume 3

Aad W Van der Vaart. 2000 · 2000

Earlier work this paper cites.

Gender differences in language use: An analysis of 14,000 text samples

Matthew L Newman, Carla J Groom, Lori D Handelman, and James W Pennebaker. 2008 · 2008

Earlier work this paper cites.

A computational approach to politeness with application to social factors

Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013 · 2013

Earlier work this paper cites.

Measuring ideological proportions in political speeches

Yanchuan Sim, Brice D. L. Acree, Justin H. Gross, and Noah A. Smith. 2013 · 2013

Earlier work this paper cites.

An attack on science? media use, trust in scientists, and perceptions of global warming

Jay D Hmielowski, Lauren Feldman, Teresa A Myers, Anthony Leiserowitz, and Edward Maibach. 2014 · 2014

Earlier work this paper cites.

XGBoost: A scalable tree boosting system

Tianqi Chen and Carlos Guestrin. 2016 · 2016

Earlier work this paper cites.

Language from police body camera footage shows racial disparities in officer respect

Rob Voigt, Nicholas P Camp, Vinodkumar Prabhakaran, William L Hamilton, Rebecca C Hetey, Camilla M Griffiths, David Jurgens, Dan Jurafsky, and Jennifer L Eberhardt. 2017 · 2017

Earlier work this paper cites.

The importance of calibration for estimating proportions from annotations

Dallas Card and Noah A Smith. 2018 · 2018

Earlier work this paper cites.

Uncertainty-aware generative models for inferring document class prevalence

Katherine Keith and Brendan O’Connor. 2018 · 2018

Earlier work this paper cites.

Media bias monitor: Quantifying biases of social media news outlets at large-scale

Filipe Ribeiro, Lucas Henrique, Fabricio Benevenuto, Abhijnan Chakraborty, Juhi Kulshrestha, Mahmoudreza Babaei, and Krishna Gummadi. 2018 · 2018

Earlier work this paper cites.

Auditing partisan audience bias within google search

Ronald E Robertson, Shan Jiang, Kenneth Joseph, Lisa Friedland, David Lazer, and Christo Wilson. 2018 · 2018

Earlier work this paper cites.

Show your work: Improved reporting of experimental results

Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith. 2019 · 2019

Earlier work this paper cites.

We can detect your bias: Predicting the political ideology of news articles

Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020 · 2020

Earlier work this paper cites.

With little power comes great responsibility

Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020 · 2020

Earlier work this paper cites.

Detecting stance in media on global warming

Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020 · 2020

Earlier work this paper cites.

Active learning by acquiring contrastive examples

Katerina Margatina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. 2021 · 2021

Earlier work this paper cites.

Causal inference in natural language processing: Estimation, prediction, interpretation and beyond

Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. 2022 · 2022

Earlier work this paper cites.

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

Earlier work this paper cites.

Annotators with attitudes: How annotator beliefs and identities bias toxic language detection

Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022 · 2022

Earlier work this paper cites.

Taxonomy of risks posed by language models

Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022 · 2022

Earlier work this paper cites.

GPT-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Cited alongside, same era.

Marked personas: Using natural language prompts to measure stereotypes in language models

Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023 · 2023

Cited alongside, same era.

Can large language models be an alternative to human evaluations?

Cheng-Han Chiang and Hung-yi Lee. 2023 · 2023

Cited alongside, same era.

When the majority is wrong: Modeling annotator disagreement for subjective tasks

Eve Fleisig, Rediet Abebe, and Dan Klein. 2023 · 2023

Cited alongside, same era.

ChatGPT outperforms crowd workers for text-annotation tasks

Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023 · 2023

Cited alongside, same era.

LLMaAA: Making large language models as active annotators

Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023 · 2023

Later among the works it cites.

Judging llm-as-a-judge with mt-bench and chatbot arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023 · 2023

Later among the works it cites.

Normbank: A knowledge bank of situational social norms

Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2023 · 2023

Later among the works it cites.

Prompt design matters for computational social science tasks but in unpredictable ways

Original

Shubham Atreja, Joshua Ashkinaze, Lingyao Li, Julia Mendelsohn, and Libby Hemphill. 2024 · 2024

Closest in time.

Can generative AI improve social science?

Christopher A Bail. 2024 · 2024

Closest in time.

Linguistic calibration of long-form generations