2023

Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks

Ullman, Tomer

Understand

Intuitive psychology is a pillar of common-sense reasoning.

The replication of this reasoning in machine intelligence is an important stepping-stone on the way to human-like artificial intelligence.
Several recent tasks and benchmarks for examining this reasoning in Large-Large Models have focused in particular on belief attribution in Theory-of-Mind tasks.
These tasks have shown both successes and failures.

Built on

Computing machinery and intelligence-AM Turing
Machinery C · 1950
Earlier work this paper cites.
Psychologism and behaviorism
Block N · 1981
Earlier work this paper cites.
Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception
Wimmer H, Perner J · 1983
Earlier work this paper cites.
Three-year-olds’ difficulty with false belief: The case for a conceptual deficit
Perner J, Leekam SR, Wimmer H · 1987
Earlier work this paper cites.
Faces in the clouds: A new theory of religion
Guthrie SE · 1995
Earlier work this paper cites.
Teleological reasoning in infancy: The naıve theory of rational action
Gergely G, Csibra G · 2003
Earlier work this paper cites.
The role of social cognition in early trust
Koenig MA, Harris PL · 2005
Earlier work this paper cites.
Secret agents: Inferences about hidden causes by 10-and 12-month-old infants
Saxe R, Tenenbaum J, Carey S · 2005
Earlier work this paper cites.
Core knowledge
Spelke ES, Kinzler KD · 2007
Earlier work this paper cites.
Action understanding as inverse planning
Baker CL, Saxe R, Tenenbaum JB · 2009
Earlier work this paper cites.
Monkeys represent others’ knowledge but not their beliefs
Marticorena DC, Ruiz AM, Mukerji C, Goddu A, Santos LR · 2011
Earlier work this paper cites.
Aristotle and the chicken: Animacy and the origins of beliefs
Vallortigara G · 2012
Earlier work this paper cites.

Similar

The happiness of the fish: Evidence for a common theory of one’s own and others’ actions
Saxe R · 2012
Cited alongside, same era.
Trust and skepticism: Children’s selective learning from testimony
Robinson EJ, Einav S · 2014
Cited alongside, same era.
The naïve utility calculus: Computational principles underlying commonsense psychology
Jara-Ettinger J, Gweon H, Schulz LE, Tenenbaum JB · 2016
Cited alongside, same era.
Six-month-old infants expect agents to minimize the cost of their actions
Liu S, Spelke ES · 2017
Cited alongside, same era.
Building machines that learn and think like people
Lake BM, Ullman TD, Tenenbaum JB, Gershman SJ · 2017
Cited alongside, same era.
Attention is all you need
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al · 2017
Cited alongside, same era.
Rational quantitative attribution of beliefs, desires and percepts in human mentalizing
Baker CL, Jara-Ettinger J, Saxe R, Tenenbaum JB · 2017
Cited alongside, same era.
Ten-month-old infants infer the value of goals from the costs of actions
Liu S, Ullman TD, Tenenbaum JB, Spelke ES · 2017
Cited alongside, same era.
Machine theory of mind
Rabinowitz N, Perbet F, Song F, Zhang C, Eslami SA, Botvinick M · 2018
Cited alongside, same era.
Evaluating theory of mind in question answering
Nematzadeh A, Burns K, Grant E, Gopnik A, Griffiths TL · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin J, Chang MW, Lee K, Toutanova K · 2018
Cited alongside, same era.
Socialiqa: Commonsense reasoning about social interactions
Sap M, Rashkin H, Chen D, LeBras R, Choi Y · 2019
Cited alongside, same era.

Then

Theory of mind as inverse reinforcement learning
Jara-Ettinger J · 2019
Later among the works it cites.
Language models are few-shot learners
Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al · 2020
Later among the works it cites.
Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others
Gandhi K, Stojnic G, Lake BM, Dillon MR · 2021
Later among the works it cites.
Agent: A benchmark for core psychological reasoning
Shu T, Bhandwaldar A, Gan C, Smith K, Liu S, Gutfreund D, et al · 2021
Later among the works it cites.
Inferential social learning: Cognitive foundations of human social learning and teaching
Gweon H · 2021
Later among the works it cites.
Neural theory-of-mind? on the limits of social intelligence in large lms
Sap M, LeBras R, Fried D, Choi Y · 2022
Later among the works it cites.
Solving quantitative reasoning problems with language models
Lewkowycz A, Andreassen A, Dohan D, Dyer E, Michalewski H, Ramasesh V, et al · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Ramesh A, Dhariwal P, Nichol A, Chu C, Chen M · 2022
Later among the works it cites.
The effects of information utility and teachers’ knowledge on evaluations of under-informative pedagogy across development
Bass I, Bonawitz E, Hawthorne-Madell D, Vong WK, Goodman ND, Gweon H · 2022
Later among the works it cites.
Testing relational understanding in text-guided image generation
Conwell C, Ullman T · 2022
Later among the works it cites.
Theory of Mind May Have Spontaneously Emerged in Large Language Models
Kosinski M · 2023
Closest in time.
Dissociating language and thought in large language models: a cognitive perspective
Mahowald K, Ivanova AA, Blank IA, Kanwisher N, Tenenbaum JB, Fedorenko E · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…