Using Large Language Models for Qualitative Analysis can Introduce Serious Bias

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. et al. (2020), ‘Language models are few-shot learners’, Advances in neural information processing systems

1901

Earlier work this paper cites.

Appadurai, A. (2004), ‘The capacity to aspire: Culture and the terms of recognition’, Culture and Public Action, ed. Vijayendra Rao and Michael Walton, Stanford, California: Stanford University Press

2004

Earlier work this paper cites.

Callard, A. (2018), Aspiration: The agency of becoming

2018

Earlier work this paper cites.

Detering, N. M. and Waters, M. (2018), ‘Flexible coding of in-depth interviews: A twenty-first century approach’, Sociological Methods and Research

2018

Earlier work this paper cites.

Edunov, S., Ott, M., Auli, M. and Grangier, D. (2018), ‘Understanding back-translation at scale’, arXiv preprint arXiv:1808.09381

Original

2018

Earlier work this paper cites.

Kearns, M. and Roth, A. (2020), The Ethical Algorithm

2020

Earlier work this paper cites.

Ashwin, J., Rao, V., Biradavolu, M., Chhabra, A., Haque, A., Krishnan, N. and Khan, A. (2022), ‘A method to scale-up interpretative qualitative analysis, with an application to aspirations in cox’s bazaar, bangladesh’

2022

Cited alongside, same era.

Bonikowski, B. and Nelson, L. K. (2022), ‘From ends to means: The promise of computational text analysis for theoretically driven sociological research’, Sociological Methods & Research

2022

Cited alongside, same era.

Mellon, J., Bailey, J., Scott, R., Breckwoldt, J. and Miori, M. (2022), ‘Does gpt-3 know what the most important issue is? using large language models to code open-text social survey responses at scale’, Using Large Language Models to Code Open-Text Social Survey Responses At Scale (December 22, 2022)

2022

Cited alongside, same era.

Small, M. L. and Calarco, J. M. (2022), Qualitative Literacy: A Guide to Evaluating Ethnographic and Interview Research

2022

Cited alongside, same era.

Dai, H., Liu, Z., Liao, W., Huang, X., Wu, Z., Zhao, L., Liu, W., Liu, N., Li, S., Zhu, D. et al. (2023), ‘Chataug: Leveraging chatgpt for text data augmentation’, arXiv preprint arXiv:2302.13007

Original

2023

Closest in time.

Gilardi, F., Alizadeh, M. and Kubli, M. (2023), ‘Chatgpt outperforms crowd-workers for text-annotation tasks’, Proceedings of the National Academy of Sciences

2023

Closest in time.

Rao, V. (2023), Can economics become more reflexive? exploring the potential of mixed-methods, in

2023

Closest in time.

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E. and Lample, G. (2023), ‘Llama: Open and efficient foundation language models’

2023

Closest in time.

Veselovsky, V., Ribeiro, M. H. and West, R. (2023), ‘Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks’, arXiv preprint arXiv:2306.07899