Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproduction

Understand

Humans extract useful abstractions of the world from noisy sensory data.

• Serial reproduction allows us to study how people construe the world through a paradigm similar to the game of telephone, where one person observes a stimulus and reproduces it for the next to form a chain of reproductions.
• Past serial reproduction experiments typically employ a single sensory modality, but humans often communicate abstractions of the world to each other through language.
• To investigate the effect language on the formation of abstractions, we implement a novel multimodal serial reproduction framework by asking people who receive a visual stimulus to reproduce it in a linguistic format, and vice versa.