Fetching the paper…
Reading the bibliography…
To what degree should we ascribe cognitive capacities to Large Language Models (LLMs), such as the ability to reason about intentions and beliefs known as Theory of Mind (ToM)? Here we add to this emerging debate by (i) testing 11 base- and instruction-tuned LLMs on capabilities relevant to ToM beyond the dominant false-belief paradigm, including non-literal language usage and recursive intentionality; (ii) using newly rewritten versions of standardized tests to gauge LLMs' robustness; (iii) prompting and scoring for open besides closed questions; and (iv) benchmarking LLM performance against that of children aged 7-10 on the same tasks.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901
Earlier work this paper cites.
Computing machinery and intelligence
Alan M. Turing. 1950 · 1950
Earlier work this paper cites.
Logic and conversation
Paul Grice. 1975 · 1975
Earlier work this paper cites.
Does the chimpanzee have a theory of mind?
David Premack and Guy Woodruff. 1978 · 1978
Earlier work this paper cites.
Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception
Heinz Wimmer and Josef Perner. 1983 · 1983
Earlier work this paper cites.
Does the autistic child have a “theory of mind”?
Simon Baron-Cohen, Alan M Leslie, and Uta Frith. 1985 · 1985
Earlier work this paper cites.
“John thinks that Mary thinks that…” attribution of second-order beliefs by 5-to 10-year-old children
Josef Perner and Heinz Wimmer. 1985 · 1985
Earlier work this paper cites.
A theory of the child’s theory of mind
J.A. Fodor. 1992 · 1992
Earlier work this paper cites.
An advanced test of theory of mind: Understanding of story characters’ thoughts and feelings by able autistic, mentally handicapped, and normal children and adults
Francesca G.E. Happé. 1994 · 1994
Earlier work this paper cites.
Theory-of-mind deficits and causal attributions
P. Kinderman, R. Dunbar, and R. P. Bentall. 1998 · 1998
Earlier work this paper cites.
The Strange Stories test - a replication study of children and adolescents with Asperger syndrome
Nils Kaland, Annette Møller-Nielsen, Lars Smith, Erik Lykke Mortensen, Kirsten Callesen, and Dorte Gottlieb. 2005 · 2005
Earlier work this paper cites.
Do 15-month-old infants understand false beliefs?
Kristine H. Onishi and Renée Baillargeon. 2005 · 2005
Earlier work this paper cites.
Language and theory of mind: Meta-analysis of the relation between language ability and false-belief understanding
Karen Milligan, Janet Wilde Astington, and Lisa Ain Dack. 2007 · 2007
Earlier work this paper cites.
Perspective-taking and memory capacity predict social network size
James Stiller and Robin IM Dunbar. 2007 · 2007
Earlier work this paper cites.
Folk Psychological Narratives: The Sociocultural Basis of Understanding Reasons
Daniel D. Hutto. 2008 · 2008
Earlier work this paper cites.
Origins of Human Communication
Michael Tomasello. 2008 · 2008
Earlier work this paper cites.
Mindreaders: the Cognitive Basis of "Theory of Mind"
Ian Apperly. 2010 · 2010
Earlier work this paper cites.
Bayesian theory of mind: Modeling joint belief-desire attribution
Chris Baker and Rebecca Saxe. 2011 · 2011
Earlier work this paper cites.
Mindreading in infancy
Peter Carruthers. 2013 · 2013
Earlier work this paper cites.
One introduction: Androids, enlightenment, and the human-machine boundary
Adelheid Voskuhl. 2013 · 2013
Earlier work this paper cites.
False belief in infancy: a fresh look
Cecilia Heyes. 2014 · 2014
Earlier work this paper cites.
The cultural evolution of mind reading
Cecilia M. Heyes and Chris D. Frith. 2014 · 2014
Earlier work this paper cites.
Grammar and cooperative communication
Arie Verhagen. 2015 · 2015
Earlier work this paper cites.
Language as a coordination tool evolves slowly
Tamas David-Barrett and Robin I. M. Dunbar. 2016 · 2016
Earlier work this paper cites.
Universal dependencies v1: A multilingual treebank collection
Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016 · 2016
Earlier work this paper cites.
The lazy mindreader: a humanities perspective on mindreading and multiple-order intentionality
Max J van Duijn. 2016 · 2016
Earlier work this paper cites.
Five-year-olds’ systematic errors in second-order false belief tasks are due to first-order theory of mind strategy selection: A computational modeling study
Burcu Arslan, Niels A Taatgen, and Rineke Verbrugge. 2017 · 2017
Earlier work this paper cites.
How can memory-augmented neural networks pass a false-belief task?
Erin Grant, Aida Nematzadeh, and Thomas L Griffiths. 2017 · 2017
Cited alongside, same era.
Higher order intentionality tasks are cognitively more demanding
Penelope A. Lewis, Amy Birch, Alexander Hall, and Robin I. M. Dunbar. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
Evaluating theory of mind in question answering
Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. 2018 · 2018
Cited alongside, same era.
Machine theory of mind
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. 2018 · 2018
Cited alongside, same era.
Infants’ performance in spontaneous-response false belief tasks: A review and meta-analysis
Pamela Barone, Guido Corradi, and Antoni Gomila. 2019 · 2019
Can AI language models replace human participants?
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023 · 2023
Closest in time.
Google’s newest A.I. model uses nearly five times more text data for training than its predecessor
Jennifer Elias. 2023 · 2023
Closest in time.
The false promise of imitating proprietary llms
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023 · 2023
Closest in time.
Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt
T. Hagendorff, S. Fabi, and M. Kosinski. 2023 · 2023
Closest in time.
Thilo Hagendorff. 2023 · 2023
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
"revisiting the evaluation of theory of mind through question answering"
Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019 · 2019
Cited alongside, same era.
Visualizing and measuring the geometry of BERT
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019 · 2019
Cited alongside, same era.
Systematic review and inventory of theory of mind measures for young children
Cindy Beaudoin, Élizabel Leblanc, Charlotte Gagner, and Miriam H. Beauchamp. 2020 · 2020
Cited alongside, same era.
Homer’s intelligent machines: AI in antiquity
Genevieve Liveley and Sam Thomas. 2020 · 2020
Cited alongside, same era.
Emergent linguistic structure in artificial neural networks trained by self-supervision
Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020 · 2020
Cited alongside, same era.
Ai-based large language models are ready to transform psychological health assessment
Oscar Kjell, Katarina Kjell, and H Andrew Schwartz. 2023 · 2023
Closest in time.
A new chip cluster will make massive ai models possible
Will Knight. 2023 · 2023
Closest in time.
Theory of mind may have spontaneously emerged in large language models
Michal Kosinski. 2023 · 2023
Closest in time.
Lost in the middle: How language models use long contexts
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023 · 2023
Closest in time.
Tomchallenges: A principle-guided dataset and diverse evaluation tasks for exploring theory of mind
Xiaomeng Ma, Lingyu Gao, and Qihui Xu. 2023 · 2023
Closest in time.
Dissociating language and thought in large language models: a cognitive perspective
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023 · 2023
Closest in time.
Developing chatgpt’s theory of mind
Antonella Marchetti, Cinzia Di Dio, Angelo Cangelosi, Federico Manzi, and Davide Massaro. 2023 · 2023
Closest in time.
Boosting theory-of-mind performance in large language models via prompting
Shima Rahimi Moghaddam and Christopher J. Honey. 2023 · 2023
Closest in time.
OpenAI. 2023 · 2023
Closest in time.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023 · 2023
Closest in time.
Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic
Billy Perrigo. 2023 · 2023
Closest in time.
Clever Hans or neural theory of mind? stress testing social reasoning in large language models
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2023 · 2023
Closest in time.
MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
Damien Sileo and Antoine Lernould. 2023 · 2023
Closest in time.
‘Full-on robot writing’: the artificial intelligence challenge facing universities
Jeff Sparrow. 2022 · 2023
Closest in time.
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros-Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2023 · 2023
Closest in time.
LLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 · 2023
Closest in time.
Large language models fail on trivial alterations to theory-of-mind tasks
Tomer Ullman. 2023 · 2023
Closest in time.
Emergent analogical reasoning in large language models
Taylor Webb, Keith J. Holyoak, and Hongjing Lu. 2023 · 2023
Closest in time.
Is recursive “mindreading” really an exception to limitations on recursive thinking?
Robert Wilson, Alexander Hruby, Daniel Perez-Zapata, Sanne W. van der Kleij, and Ian A. Apperly. 2023 · 2023
Closest in time.
A comprehensive capability analysis of GPT-3 and GPT-3.5 series models
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023 · 2023
Closest in time.
LIMA: Less is more for alignment
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023 · 2023
Closest in time.