Diminished Diversity-of-Thought in a Standard Large Language Model

Understand

We test whether Large Language Models (LLMs) can be used to simulate human participants in social-science studies.

• To do this, we run replications of 14 studies from the Many Labs 2 replication project with OpenAI's text-davinci-003 model, colloquially known as GPT3.5.
• Based on our pre-registered analyses, we find that among the eight studies we could analyse, our GPT sample replicated 37.5% of the original results and 37.5% of the Many Labs 2 results.
• However, we were unable to analyse the remaining six studies due to an unexpected phenomenon we call the "correct answer" effect.