Automated Annotation with Generative AI Requires Validation

Language models are few-shot learners

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020) · 1901

Earlier work this paper cites.

A template approach to text analysis: Developing and using codebooks

Crabtree, B. F. and Miller, W. L. (1992) · 1992

Earlier work this paper cites.

Codebook development for team-based qualitative analysis

MacQueen, K. M., McLellan, E., Kay, K., and Milstein, B. (1998) · 1998

Earlier work this paper cites.

Text as data: The promise and pitfalls of automatic content analysis methods for political texts

Grimmer, J. and Stewart, B. M. (2013) · 2013

Earlier work this paper cites.

The Content Analysis Guidebook

Neuendorf, K. A. (2016) · 2016

Earlier work this paper cites.

Content Analysis: An Introduction to Its Methodology

Krippendorff, K. (2018) · 2018

Earlier work this paper cites.

An mturk crisis? shifts in data quality and the impact on study results

Chmielewski, M. and Kucker, S. C. (2020) · 2020

Earlier work this paper cites.

People judge others to have more voluntary control over beliefs than they themselves do

Cusimano, C. and Goodwin, G. P. (2020) · 2020

Earlier work this paper cites.

Want to reduce labeling cost? GPT-3 can help

Wang, S., Liu, Y., Xu, Y., Zhu, C., and Zeng, M. (2021) · 2021

Cited alongside, same era.

Computational analysis of 140 years of us political speeches reveals more positive but increasingly polarized framing of immigration

Card, D., Chang, S., Becker, C., Mendelsohn, J., Voigt, R., Boustan, L., Abramitzky, R., and Jurafsky, D. (2022) · 2022

Cited alongside, same era.

Is gpt-3 a good data annotator?

Ding, B., Qin, C., Liu, L., Bing, L., Joty, S., and Li, B. (2022) · 2022

Cited alongside, same era.

Introducing the gab hate corpus: defining and applying hate-based rhetoric to social media posts at scale

Kennedy, B., Atari, M., Davani, A. M., Yeh, L., Omrani, A., Kim, Y., Coombs, K., Havaldar, S., Portillo-Wightman, G., Gonzalez, E., Hoover, J., Azatian, A., Hussain, A., Lara, A., Olmos, G., Omary, A., Park, C., Wijaya, C., Wang, X., Zhang, Y., and Dehghani, M. (2022) · 2022

Cited alongside, same era.

Informing the leader: Bureaucracies and international crises

Schub, R. (2022) · 2022

Cited alongside, same era.

Chatgpt outperforms crowd-workers for text-annotation tasks

Gilardi, F., Alizadeh, M., and Kubli, M. (2023) · 2023

Closest in time.

Annollm: Making large language models to be better crowdsourced annotators

He, X., Lin, Z., Gong, Y., Jin, A.-L., Zhang, H., Lin, C., Jiao, J., Yiu, S. M., Duan, N., and Chen, W. (2023) · 2023

Closest in time.

Testing the reliability of chatgpt for text annotation and classification: A cautionary remark

Reiss, M. (2023) · 2023

Closest in time.

On the rise of fear speech in online social media

Saha, P., Narla, Kalyan, K., and Mukherjee, A. (2023) · 2023

Closest in time.

Why open-source generative ai models are an ethical way forward for science

Spirling, A. (2023) · 2023

Closest in time.

Can chatgpt reproduce human-generated labels? a study of social computing tasks

Zhu, Y., Zhang, P., Haq, E.-U., Hui, P., and Tyson, G. (2023) · 2023