2022

What Artificial Neural Networks Can Tell Us About Human Language Acquisition

Warstadt, Alex, Bowman, Samuel R.

Understand

Rapid progress in machine learning for natural language processing has the potential to transform debates about how humans learn language.

However, the learning environments and biases of current artificial learners and humans diverge in ways that weaken the impact of the evidence obtained from learning simulations.
For example, today's most effective neural language models are trained on roughly one thousand times the amount of linguistic data available to a typical child.
To increase the relevance of learnability results from computational models, we need to train model learners without significant advantages over humans.

Built on

RoBERTa: A robustly optimized BERT pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 1907
Earlier work this paper cites.
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
Original
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet · 1912
Earlier work this paper cites.
How to Do Things With Words
JL Austin · 1962
Earlier work this paper cites.
Aspects of the Theory of Syntax
Noam Chomsky · 1965
Earlier work this paper cites.
Language identification in the limit
E Mark Gold · 1967
Earlier work this paper cites.
Speech acts: An essay in the philosophy of language , volume 626
John R. Searle · 1969
Earlier work this paper cites.
Problems of knowledge and freedom: The Russell lectures
Noam Chomsky · 1971
Earlier work this paper cites.
The Princes of the World
G. G. Coulton · 1972
Earlier work this paper cites.
A theory of the learnable
L. G. Valiant · 1972
Earlier work this paper cites.
The Formal Theory of Grammar
John P. Kimball · 1973
Earlier work this paper cites.
The proper treatment of quantification in ordinary English
Richard Montague · 1973
Earlier work this paper cites.
The Development of Language in Genie: a Case of Language Acquisition beyond the ”Critical Period”
Victoria Fromkin, Stephen Krashen, Susan Curtiss, David Rigler, and Marilyn Rigler · 1974
Earlier work this paper cites.
Introduction to Generative-Transformational Synta
Carl Lee Baker · 1978
Earlier work this paper cites.
The need for biases in learning generalizations
Tom M Mitchell · 1980
Earlier work this paper cites.
Why nouns are learned before verbs: Linguistic relativity versus natural partitioning
Dedre Gentner · 1982
Earlier work this paper cites.
Language Acquisition: The State of the Art
Lila Gleitman and Eric Wanner · 1982
Earlier work this paper cites.
A Generative Theory of Tonal Music
Fred Lerdahl, Ray S Jackendoff, and Ray Jackendoff · 1983
Earlier work this paper cites.
Level-ordering in lexical development
Peter Gordon · 1985
Earlier work this paper cites.
Structure Dependence in Grammar Formation
Stephen Crain and Mineharu Nakayama · 1987
Earlier work this paper cites.
Estimation of probabilities from sparse data for the language model component of a speech recognizer
Slava Katz · 1987
Earlier work this paper cites.
Connectionism and cognitive architecture: A critical analysis
Jerry A Fodor and Zenon W Pylyshyn · 1988
Earlier work this paper cites.
Finding structure in time
Jeffrey L Elman · 1990
Earlier work this paper cites.
Probably approximately correct learning
David Haussler · 1990
Earlier work this paper cites.
The theory of principles and parameters
Noam Chomsky and Howard Lasnik · 1993
Earlier work this paper cites.
American parenting of language-learning children: Persisting differences in family-child interactions observed in natural home environments
Betty Hart and Todd R. Risley · 1993
Earlier work this paper cites.
Negative evidence in language acquisition
Gary F Marcus · 1993
Earlier work this paper cites.
The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology
Carson T. Schütze · 1996
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
A Solution to Plato’s Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge
Thomas K Landauer and Susan T Dutnais · 1997
Earlier work this paper cites.
Function, selection, and innateness: The emergence of language universals
Simon Kirby · 1999
Earlier work this paper cites.
Infant artificial language learning and language acquisition
Rebecca L Gómez and LouAnn Gerken · 2000
Earlier work this paper cites.
A Probabilistic Earley Parser as a Psycholinguistic Model
John Hale · 2001
Earlier work this paper cites.
Understanding stimulus poverty arguments
Janet Dean Fodor and Carrie Crowther · 2002
Earlier work this paper cites.
Empirical re-assessment of stimulus poverty arguments
Julie Anne Legate and Charles D Yang · 2002
Earlier work this paper cites.
Adult reformulations of child errors as negative evidence
Michelle M Chouinard and Eve V Clark · 2003
Earlier work this paper cites.
What infants know about syntax but couldn’t have learned: Experimental evidence for syntactic structure at 18 months
Jeffrey Lidz, Sandra Waxman, and Jennifer Freedman · 2003
Earlier work this paper cites.
The prosodic bootstrapping of phrases: Evidence from prelinguistic infants
Melanie Soderstrom, Amanda Seidl, Deborah G Kemler Nelson, and Peter W Jusczyk · 2003
Earlier work this paper cites.
A model of grounded language acquisition: Sensorimotor features improve lexical and grammatical learning
Steve R. Howell, Damian Jankowicz, and Suzanna Becker · 2005
Earlier work this paper cites.
Uncovering the richness of the stimulus: Structure dependence and indirect statistical evidence
Florencia Reali and Morten H Christiansen · 2005
Earlier work this paper cites.
Emergent Multi-Agent Communication in the Deep Learning Era
Original
Angeliki Lazaridou and Marco Baroni · 2006
Earlier work this paper cites.
Learning phonology with substantive bias: An experimental and computational study of velar palatalization
Colin Wilson · 2006
Earlier work this paper cites.
Biolinguistic Explorations: Design, Development, Evolution
Noam Chomsky · 2007
Earlier work this paper cites.
Transformational networks
Robert Frank and Donald Mathis · 2007
Earlier work this paper cites.
Expectation-based syntactic comprehension
Roger Levy · 2007
Earlier work this paper cites.
The 385+ million word Corpus of Contemporary American English (1990–2008+): Design, architecture, and linguistic insights
Mark Davies · 2009
Earlier work this paper cites.
Poverty of the stimulus revisited
Robert C Berwick, Paul Pietroski, Beracah Yankama, and Noam Chomsky · 2011
Earlier work this paper cites.
Linguistic Nativism and the Poverty of the Stimulus
Alexander Clark and Shalom Lappin · 2011
Earlier work this paper cites.
The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling
Original
Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux · 2011
Earlier work this paper cites.
The learnability of abstract syntactic principles
Andy Perfors, Joshua B Tenenbaum, and Terry Regier · 2011
Earlier work this paper cites.
A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010
Jon Sprouse, Carson T. Schütze, and Diogo Almeida · 2013
Earlier work this paper cites.
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Original
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio · 2014
Earlier work this paper cites.
The CHILDES project: Tools for analyzing talk, Volume II: The database
Brian MacWhinney · 2014
Earlier work this paper cites.
Microsoft COCO Captions: Data Collection and Evaluation Server
Original
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick · 2015
Earlier work this paper cites.
Semi-supervised sequence learning
Andrew M Dai and Quoc V Le · 2015
Earlier work this paper cites.

Similar

Combining Language and Vision with a Multimodal Skip-gram Model
Angeliki Lazaridou, Nghia The Pham, and Marco Baroni · 2015
Cited alongside, same era.
Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton · 2015
Cited alongside, same era.
Creating language: Integrating evolution, acquisition, and processing
Morten H Christiansen and Nick Chater · 2016
Cited alongside, same era.
Probing for semantic evidence of composition by means of simple classification tasks
Allyson Ettinger, Ahmed Elgohary, and Philip Resnik · 2016
Cited alongside, same era.
Endogenous sources of variation in language acquisition
Chung-hye Han, Julien Musolino, and Jeffrey Lidz · 2016
Cited alongside, same era.
Assessing the ability of LSTMs to learn syntax-sensitive dependencies
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg · 2016
Cited alongside, same era.
OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles
Pierre Lison and Jorg Tiedemann · 2016
Cited alongside, same era.
Grammar of binding in the languages of the world: Unity versus diversity
Eric Reuland · 2016
Cited alongside, same era.
Does string-based neural MT learn source syntax?
Xing Shi, Inkit Padhi, and Kevin Knight · 2016
Cited alongside, same era.
Bootstrapping language acquisition
Omri Abend, Tom Kwiatkowski, Nathaniel J. Smith, Sharon Goldwater, and Mark Steedman · 2017
Cited alongside, same era.
Fine-grained analysis of sentence embeddings using auxiliary prediction tasks
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg · 2017
Cited alongside, same era.
Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner
Original
Emmanuel Dupoux · 2017
Cited alongside, same era.

Then

BERT has a mouth, and it must speak: BERT as a Markov random field language model
Alex Wang and Kyunghyun Cho · 2019
Later among the works it cites.
Linguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgments
Original
Alex Warstadt and Samuel R. Bowman · 2019
Later among the works it cites.
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Later among the works it cites.
What Don’t RNN Language Models Learn About Filler-Gap Dependencies?
Rui P. Chaves · 2020
Later among the works it cites.
UNITER: UNiversal image-text representation learning
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu · 2020
Later among the works it cites.
SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy · 2020
Later among the works it cites.
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen · 2020
Later among the works it cites.
A Systematic Assessment of Syntactic Generalization in Neural Language Models
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy · 2020
Later among the works it cites.
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
Najoung Kim and Tal Linzen · 2020
Later among the works it cites.
Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman · 2020
Later among the works it cites.
Emergent linguistic structure in artificial neural networks trained by self-supervision
Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy · 2020
Later among the works it cites.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
R. Thomas McCoy, Robert Frank, and Tal Linzen · 2020
Later among the works it cites.
Recurrent babbling: evaluating the acquisition of grammar from limited input data
Ludovica Pannitto and Aurélie Herbelot · 2020
Later among the works it cites.
Information-Theoretic Probing for Linguistic Structure
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell · 2020
Later among the works it cites.
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg · 2020
Later among the works it cites.
Capacity, bandwidth, and compositionality in emergent language learning
Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew M Dai, and Kyunghyun Cho · 2020
Later among the works it cites.
“LazImpa”: Lazy and Impatient neural agents learn to communicate efficiently
Mathieu Rita, Rahma Chaabouni, and Emmanuel Dupoux · 2020
Later among the works it cites.
A Primer in BERTology: What we know about how BERT works
Anna Rogers, Olga Kovaleva, and Anna Rumshisky · 2020
Later among the works it cites.
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi · 2020
Later among the works it cites.
Masked Language Model Scoring
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff · 2020
Later among the works it cites.
Vl-bert: Pre-training of generic visual-linguistic representations
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai · 2020
Later among the works it cites.
Investigating gender bias in language models using causal mediation analysis
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber · 2020
Later among the works it cites.
Information-Theoretic Probing with Minimum Description Length
Elena Voita and Ivan Titov · 2020
Later among the works it cites.
Can neural networks acquire a structural bias from raw linguistic data?
Alex Warstadt and Samuel R Bowman · 2020
Later among the works it cites.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Alex Warstadt, Yian Zhang, Haau-Sing Li, Haokun Liu, and Samuel R Bowman · 2020
Later among the works it cites.
Bayesian deep learning and a probabilistic perspective of generalization
Andrew G Wilson and Pavel Izmailov · 2020
Later among the works it cites.
On the proper role of linguistically-oriented deep net analysis in linguistic theorizing
Original
Marco Baroni · 2021
Later among the works it cites.
The grammar-learning trajectories of neural language models
Original
Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend · 2021
Later among the works it cites.
(What) Can Deep Learning Contribute to Theoretical Linguistics?
Gabe Dupre · 2021
Later among the works it cites.
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov · 2021
Later among the works it cites.
The Natural Stories corpus: a reading-time corpus of English texts containing rare syntactic constructions
Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven T. Piantadosi, and Evelina Fedorenko · 2021
Later among the works it cites.
Using lexical context to discover the noun category: Younger children have it easier
Philip A. Huebner and Jon A. Willits · 2021
Later among the works it cites.
Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models
Taichi Iki and Akiko Aizawa · 2021
Later among the works it cites.
MDETR-modulated detection for end-to-end multi-modal understanding
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion · 2021
Later among the works it cites.
Generative Spoken Language Modeling from Raw Audio
Original
Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, and Emmanuel Dupoux · 2021
Later among the works it cites.
Syntactic Structure from Deep Learning
Tal Linzen and Marco Baroni · 2021
Later among the works it cites.
Predicting Inductive Biases of Fine-tuned Models
Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick · 2021
Later among the works it cites.
Deep subjecthood: Higher-order grammatical features in multilingual BERT
Isabel Papadimitriou, Ethan A. Chi, Richard Futrell, and Kyle Mahowald · 2021
Later among the works it cites.
Transformers Generalize Linearly
Original
Jackson Petty and Robert Frank · 2021
Later among the works it cites.
How much pretraining data do language models need to learn syntax?
Original
Laura Pérez-Mayos, Miguel Ballesteros, and Leo Wanner · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Original
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and others · 2021
Later among the works it cites.
Scaling language models: Methods, analysis & insights from training gopher
Original
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, and others · 2021
Later among the works it cites.
The nature of the semantic stimulus: the acquisition of every as a case study
Ezer Rasin and Athulya Aravind · 2021
Later among the works it cites.
Flava: A foundational language and vision alignment model
Original
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela · 2021
Later among the works it cites.
SAYCam: A Large, Longitudinal Audiovisual Dataset Recorded From the Infant’s Perspective
Jessica Sullivan, Michelle Mei, Andrew Perfors, Erica Wojcik, and Michael C. Frank · 2021
Later among the works it cites.
A Targeted Assessment of Incremental Processing in Neural Language Models and Humans
Ethan Wilcox, Pranali Vani, and Roger Levy · 2021
Later among the works it cites.
Does Vision-and-Language Pretraining Improve Lexical Grounding?
Tian Yun, Chen Sun, and Ellie Pavlick · 2021
Later among the works it cites.
When Do You Need Billions of Words of Pretraining Data?
Original
Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman · 2021
Later among the works it cites.
Emergent communication at scale
Rahma Chaabouni, Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot · 2022
Closest in time.
Word Acquisition in Neural Language Models
Tyler A. Chang and Benjamin K. Bergen · 2022
Closest in time.
PaLM: Scaling language modeling with pathways
Original
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and others · 2022
Closest in time.
Early phonetic learning from ecological audio: domain-general versus domain-specific mechanisms
Marvin Lavechin, Maureen de Seyssel, Marianne Métais, Florian Metze, Abdelrahman Mohamed, Hervé BREDIN, Emmanuel Dupoux, and Alejandrina Cristia · 2022
Closest in time.
One model for the learning of language
Yuan Yang and Steven T. Piantadosi · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…