Are Chatbots Reliable Text Annotators? Sometimes

Understand

Recent research highlights the significant potential of ChatGPT for text annotation in social science research.

• However, ChatGPT is a closed-source product which has major drawbacks with regards to transparency, reproducibility, cost, and data protection.
• Recent advances in open-source (OS) large language models (LLMs) offer an alternative without these drawbacks.
• Thus, it is important to evaluate the performance of OS LLMs relative to ChatGPT and standard approaches to supervised machine learning classification.