Memory Is All You Need: Testing How Model Memory Affects LLM Performance in Annotation Tasks

“Deep reinforcement learning from human preferences.” Advances in neural information processing systems

Christiano, Paul F, Jan Leike, Tom Brown, Miljan Martic, Shane Legg and Dario Amodei. 2017 · 2017

Earlier work this paper cites.

“Politicians in the line of fire: Incivility and the treatment of women on social media.” Research & Politics

Rheault, Ludovic, Erica Rayment and Andreea Musulan. 2019 · 2019

Earlier work this paper cites.

“Training language models to follow instructions with human feedback.” Advances in neural information processing systems

Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray et al. 2022 · 2022

Earlier work this paper cites.

“Chain-of-thought prompting elicits reasoning in large language models.” Advances in neural information processing systems

Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou et al. 2022 · 2022

Earlier work this paper cites.

“Gpt-4 technical report.” arXiv preprint arXiv:2303.08774

Original

Achiam, Josh, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat et al. 2023 · 2023

Earlier work this paper cites.

“Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks.” arXiv preprint arXiv:2307.02179

Original

Alizadeh, Meysam, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikova and Fabrizio Gilardi. 2023 · 2023

Earlier work this paper cites.

“When do you need chain-of-thought prompting for chatgpt?” arXiv preprint arXiv:2304.03262

Original

Chen, Jiuhai, Lichang Chen, Heng Huang and Tianyi Zhou. 2023 · 2023

Earlier work this paper cites.

“Contrastive chain-of-thought prompting.” arXiv preprint arXiv:2311.09277

Original

Chia, Yew Ken, Guizhen Chen, Luu Anh Tuan, Soujanya Poria and Lidong Bing. 2023 · 2023

Earlier work this paper cites.

“ChatGPT outperforms crowd workers for text-annotation tasks.” Proceedings of the National Academy of Sciences

Gilardi, Fabrizio, Meysam Alizadeh and Maël Kubli. 2023 · 2023

Earlier work this paper cites.

Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech. In Companion Proceedings of the ACM Web Conference 2023

Huang, Fan, Haewoon Kwak and Jisun An. 2023 · 2023

Earlier work this paper cites.

“ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification.”

Kuzman, Taja, Igor Mozetič and Nikola Ljubešić. 2023 · 2023

Earlier work this paper cites.

“CQSumDP: a ChatGPT-annotated resource for query-focused abstractive summarization based on debatepedia.” arXiv preprint arXiv:2305.06147

Original

Laskar, Md Tahmid Rahman, Mizanur Rahman, Israt Jahan, Enamul Hoque and Jimmy Huang. 2023 · 2023

Cited alongside, same era.

“Uncovering the semantics of concepts using GPT-4.” Proceedings of the National Academy of Sciences

Le Mens, Gaël, Balázs Kovács, Michael T Hannan and Guillem Pros. 2023 · 2023

Cited alongside, same era.

“How to train your stochastic parrot: Large language models for political texts.” Political Science Research and Methods

Ornstein, Joseph T, Elise N Blasingame and Jake S Truscott. 2023 · 2023

Cited alongside, same era.

Chatgpt vs. crowdsourcing vs. experts: Annotating open-domain conversations with speech functions. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue

Ostyakova, Lidiia, Veronika Smilga, Kseniia Petukhova, Maria Molchanova and Daniel Kornev. 2023 · 2023

Cited alongside, same era.

“Nostalgia in European party politics: a text-based measurement approach.” British Journal of Political Science

Müller, Stefan and Sven-Oliver Proksch. 2024 · 2024

Later among the works it cites.

Human vs ChatGPT: Effect of Data Annotation in Interpretable Crisis-Related Microblog Classification. In Proceedings of the ACM on Web Conference 2024

Nguyen, Thi Huyen and Koustav Rudra. 2024 · 2024

Later among the works it cites.

“Coding with the machines: machine-assisted coding of rare event data.” PNAS nexus

Overos, Henry David, Roman Hlatky, Ojashwi Pathak, Harriet Goers, Jordan Gouws-Dewar, Katy Smith, Keith Padraic Chew, Jóhanna K Birnir and Amy H Liu. 2024 · 2024

Later among the works it cites.

“Using proprietary language models in academic research requires explicit justification.” Nature Computational Science

Palmer, Alexis, Noah A Smith and Arthur Spirling. 2024 · 2024

Later among the works it cites.

“GPT is an effective tool for multilingual psychological text analysis.” Proceedings of the National Academy of Sciences