Out of One, Many: Using Language Models to Simulate Human Samples

Understand

We propose and explore the possibility that language models can be studied as effective proxies for specific human sub-populations in social science research.

• Practical and research applications of artificial intelligence tools have sometimes been limited by problematic biases (such as racism or sexism), which are often treated as uniform properties of the models.
• We show that the "algorithmic bias" within one such tool -- the GPT-3 language model -- is instead both fine-grained and demographically correlated, meaning that proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups.
• We term this property "algorithmic fidelity" and explore its extent in GPT-3.