2020

Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration

Colas, Cédric, Karch, Tristan, Lair, Nicolas et al.

Understand

Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills.

Such agents need to create and represent goals, select which ones to pursue and learn to achieve them.
Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states.
This limited agents to sample goals within the distribution of known effects.

Built on

Using Natural Language for Reward Shaping in Reinforcement Learning
Original
Prasoon Goyal, Scott Niekum, and Raymond J. Mooney · 1903
Earlier work this paper cites.
A hitchhiker’s guide to statistical comparisons of reinforcement learning algorithms
Original
Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer · 1904
Earlier work this paper cites.
Language as an Abstraction for Hierarchical Deep Reinforcement Learning
Original
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn · 1906
Earlier work this paper cites.
A Survey of Reinforcement Learning Informed by Natural Language
Original
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel · 1906
Earlier work this paper cites.
The language and thought of the child
Jean Piaget · 1926
Earlier work this paper cites.
Syntactic structures
Noam. Chomsky · 1957
Earlier work this paper cites.
Tool and Symbol in Child Development
L. S. Vygotsky · 1978
Earlier work this paper cites.
The Narrative Construction of Reality
Jerome Bruner · 1991
Earlier work this paper cites.
Maternal responsiveness to infants in three societies: The united states, france, and japan
Marc H Bornstein, Catherine S Tamis-LeMonda, Joseph Tal, Pamela Ludemann, Sueko Toda, Charles W Rahn, Marie-Germaine Pêcheux, Hiroshi Azuma, and Danya Vardi · 1992
Earlier work this paper cites.
Origins of knowledge
Elizabeth S Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson · 1992
Earlier work this paper cites.
Twenty-three-month-old children have a grammatical category of noun
Michael Tomasello and Raquel Olguin · 1993
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
The scientist in the crib: Minds, brains, and how children learn
Alison Gopnik, Andrew N Meltzoff, and Patricia K Kuhl · 1999
Earlier work this paper cites.
The cultural origins of human cognition
Michael Tomasello · 1999
Earlier work this paper cites.
The item-based nature of children’s early syntactic development
Michael Tomasello · 2000
Earlier work this paper cites.
Grounding language in action
Arthur M. Glenberg and Michael P. Kaschak · 2002
Earlier work this paper cites.
Constructions: A new theoretical approach to language
Adele E Goldberg · 2003
Earlier work this paper cites.
Development of object concepts in infancy: Evidence for early learning in an eye-tracking paradigm
Scott P Johnson, Dima Amso, and Jonathan A Slemmer · 2003
Earlier work this paper cites.
Frequent frames as a cue for grammatical categories in child directed speech
Toben H Mintz · 2003
Earlier work this paper cites.
Intrinsically motivated reinforcement learning
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh · 2005
Earlier work this paper cites.
Emergence of grammatical constructions: evidence from simulation and grounded agent experiments
Peter Ford Dominey · 2005
Earlier work this paper cites.
Embodied sentence comprehension
Rolf Zwaan and Carol Madden · 2005
Earlier work this paper cites.
Semiotic dynamics for embodied agents
Luc Steels · 2006
Earlier work this paper cites.
In search of the neural circuits of intrinsic motivation
Frederic Kaplan and Pierre-Yves Oudeyer · 2007
Earlier work this paper cites.
Intrinsic motivation systems for autonomous mental development
Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V Hafner · 2007
Earlier work this paper cites.
Cognitive developmental robotics: A survey
Minoru Asada, Koh Hosoda, Yasuo Kuniyoshi, Hiroshi Ishiguro, Toshio Inui, Yuichiro Yoshikawa, Masaki Ogino, and Chisato Yoshida · 2009
Earlier work this paper cites.

Similar

A cognitive neuroscience perspective on embodied language for human–robot cooperation
Carol Madden, Michel Hoen, and Peter Ford Dominey · 2009
Cited alongside, same era.
Constructing a language
Michael Tomasello · 2009
Cited alongside, same era.
Reading Between the Lines: Learning to Map High-level Instructions to Commands
S R. K. Branavan, Luke S. Zettlemoyer, and Regina Barzilay · 2010
Cited alongside, same era.
Formal theory of creativity, fun, and intrinsic motivation (1990–2010)
Jürgen Schmidhuber · 2010
Cited alongside, same era.
Learning to Interpret Natural Language Navigation Instructions from Observations
David L. Chen and Raymond J. Mooney · 2011
Cited alongside, same era.
Intrinsically motivated learning in natural and artificial systems
Gianluca Baldassarre and Marco Mirolli · 2013
Cited alongside, same era.
Active learning of inverse models with intrinsically motivated goal exploration in robots
Adrien Baranes and Pierre-Yves Oudeyer · 2013
Cited alongside, same era.
Real-time parallel processing of grammatical structure in the fronto-striatal system: A recurrent network simulation study using reservoir computing
Xavier Hinaut and Peter Ford Dominey · 2013
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Cited alongside, same era.
Recurrent neural network regularization, 2014
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals · 2014
Cited alongside, same era.
Developmental robotics: From babies to robots
Angelo Cangelosi and Matthew Schlesinger · 2015
Cited alongside, same era.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Cited alongside, same era.

Then

Off-policy deep reinforcement learning without exploration
Original
Scott Fujimoto, David Meger, and Doina Precup · 2018
Later among the works it cites.
Curiosity driven exploration of learned disentangled goal spaces
Adrien Laversanne-Finot, Alexandre Pere, and Pierre-Yves Oudeyer · 2018
Later among the works it cites.
Unicorn: Continual learning with a universal, off-policy agent
Original
Daniel J. Mankowitz, Augustin Zídek, André Barreto, Dan Horgan, Matteo Hessel, John Quan, Junhyuk Oh, Hado van Hasselt, David Silver, and Tom Schaul · 2018
Later among the works it cites.
Visual reinforcement learning with imagined goals
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine · 2018
Later among the works it cites.
Multi-goal reinforcement learning: Challenging robotics environments and request for research
Original
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al · 2018
Later among the works it cites.
Monet: Unsupervised scene decomposition and representation
Original
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner · 2019
Later among the works it cites.
Actrce: Augmenting experience via teacher’s advice for multi-goal reinforcement learning, 2019
Harris Chan, Yuhuai Wu, Jamie Kiros, Sanja Fidler, and Jimmy Ba · 2019
Later among the works it cites.
Baby{AI}: First Steps Towards Grounded Language Learning With a Human In the Loop
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio · 2019
Later among the works it cites.
Self-educated language agent with hindsight experience replay for instruction following
Original
Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin · 2019
Later among the works it cites.
CURIOUS: intrinsically motivated modular multi-goal reinforcement learning
Cédric Colas, Pierre-Yves Oudeyer, Olivier Sigaud, Pierre Fournier, and Mohamed Chetouani · 2019
Later among the works it cites.
Go-explore: a new approach for hard-exploration problems
Original
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune · 2019
Later among the works it cites.
From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama · 2019
Later among the works it cites.
Multi-object representation learning with iterative variational inference
Original
Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner · 2019
Later among the works it cites.
Emergent systematic generalization in a situated agent, 2019
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L. McClelland, and Adam Santoro · 2019
Later among the works it cites.
Measuring compositional generalization: A comprehensive method on realistic data, 2019
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet · 2019
Later among the works it cites.
Extending machine language models toward human-level language understanding
Original
James L McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze · 2019
Later among the works it cites.
Contextual imagined goals for self-supervised robotic learning
Original
Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine · 2019
Later among the works it cites.
Vision-based navigation with language-based assistance via imitation learning with indirect intervention
Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan · 2019
Later among the works it cites.
Skew-fit: State-covering self-supervised reinforcement learning
Original
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine · 2019
Later among the works it cites.
Automated curricula through setter-solver interactions
Original
Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Later among the works it cites.
Self-supervised learning of distance functions for goal-conditioned reinforcement learning
Original
Srinivas Venkattaramanujam, Eric Crawford, Thang Doan, and Doina Precup · 2019
Later among the works it cites.
Good-enough compositional data augmentation
Jacob Andreas · 2020
Closest in time.
Exploratory play, rational action, and efficient search
Junyi Chu and Laura Schulz · 2020
Closest in time.
Deep sets for generalization in rl, 2020
Tristan Karch, Cédric Colas, Laetitia Teodorescu, Clément Moulin-Frier, and Pierre-Yves Oudeyer · 2020
Closest in time.
Alfred: A benchmark for interpreting grounded instructions for everyday tasks
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…