2018

Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

Péré, Alexandre, Forestier, Sébastien, Sigaud, Olivier et al.

Understand

Intrinsically motivated goal exploration algorithms enable machines to discover repertoires of policies that produce a diversity of effects in complex environments.

These exploration algorithms have been shown to allow real world robots to acquire skills such as tool use in high-dimensional continuous state and action spaces.
However, they have so far assumed that self-generated goals are sampled in a specifically engineered feature space, limiting their autonomy.
In this work, we propose to use deep representation learning algorithms to learn an adequate goal space.

Built on

On lines and planes of closest fit to systems of points in space
Karl Pearson · 1941
Earlier work this paper cites.
Remarks on Some Nonparametric Estimates of a Density Function
Murray Rosenblatt · 1956
Earlier work this paper cites.
On Estimation of a Probability Density Function and Mode
Emanuel Parzen · 1962
Earlier work this paper cites.
Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis
Joseph B. Kruskal · 1964
Earlier work this paper cites.
Curiosity and exploration
Daniel E. Berlyne · 1966
Earlier work this paper cites.
A teachable machine in the real world
Peter M Andreae and John H. Andreae · 1978
Earlier work this paper cites.
Auto-association by multilayer perceptrons and singular value decomposition
Hervé Bourlard and Yves Kamp · 1988
Earlier work this paper cites.
Integrated architectures for learning, planning, and reacting based on approximating dynamic programming
Richard S. Sutton · 1990
Earlier work this paper cites.
A possibility for implementing curiosity and boredom in model-building neural controllers
Jurgen Schmidhuber · 1991
Earlier work this paper cites.
Multivariate Density Estimation: Theory, Practice, and Visualization , volume 156
David W. Scott · 1992
Earlier work this paper cites.
Online Algorithms and Stochastic Approximations
Leon Bottou · 1998
Earlier work this paper cites.
The scientist in the crib: Minds, brains, and how children learn
Alison Gopnik, Andrew N. Meltzoff, and Patricia K. Kuhl · 1999
Earlier work this paper cites.
A global geometric framework for nonlinear dimensionality reduction
Joshua B. Tenenbaum, Veronica De Silva, and John C. Langford · 2000
Earlier work this paper cites.
Motivational principles for visual know-how development
Frederic Kaplan and Pierre-Yves Oudeyer · 2003
Earlier work this paper cites.
Computational approaches to motor learning by imitation
Stefan Schaal, Auke Ijspeert, and Aude Billard · 2003
Earlier work this paper cites.
An action perspective on motor development
Claes Von Hofsten · 2004
Earlier work this paper cites.
Intrinsically motivated reinforcement learning
Nuttapong Chentanez, Andrew G Barto, and Satinder P. Singh · 2005
Earlier work this paper cites.
Intrinsic motivation systems for autonomous mental development
Pierre-Yves Oudeyer, Frederic Kaplan, and Verena V. Hafner · 2006
Earlier work this paper cites.

Similar

What is intrinsic motivation? A typology of computational approaches
Pierre-Yves Oudeyer and Frederic Kaplan · 2007
Cited alongside, same era.
R-IAC: Robust intrinsically motivated exploration and active learning
Adrien Baranes and Pierre-Yves Oudeyer · 2009
Cited alongside, same era.
Goal babbling permits direct learning of inverse kinematics
M. Rolf, Jochen J. Steil, and Michael Gienger · 2010
Cited alongside, same era.
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
John Duchi, Elad Hazan, and Yoram Singer · 2012
Cited alongside, same era.
Intrinsically motivated learning in natural and artificial systems
Gianluca Baldassarre, Marco Mirolli, and Andrew G. Barto · 2013
Cited alongside, same era.
Active learning of inverse models with intrinsically motivated goal exploration in robots
Adrien Baranes and Pierre-Yves Oudeyer · 2013
Cited alongside, same era.
Intrinsic motivation and reinforcement learning
Andrew G. Barto · 2013
Cited alongside, same era.
Dynamical movement primitives: learning attractor models for motor behaviors
Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal · 2013
Cited alongside, same era.
Auto-encoding variational bayes, 2013
Original
Diederik P. Kingma and Max Welling · 2013
Cited alongside, same era.
Learning and exploration in action-perception loops
Daniel Y Little and Friedrich T Sommer · 2013
Cited alongside, same era.
Information driven self-organization of complex robotic behaviors
Georg Martius, Ralf Der, and Nihat Ay · 2013
Cited alongside, same era.
Intrinsically motivated learning of real-world sensorimotor skills with developmental constraints
Pierre-Yves Oudeyer, Adrien Baranes, and Frederic Kaplan · 2013
Cited alongside, same era.

Then

Variational inference with normalizing flows
Danilo J. Rezende and Shakir Mohamed · 2015
Later among the works it cites.
Why greatness cannot be planned: The myth of the objective
Kenneth O. Stanley and Joel Lehman · 2015
Later among the works it cites.
Unifying count-based exploration and intrinsic motivation
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos · 2016
Later among the works it cites.
Modular active curiosity-driven discovery of tool use
Sébastien Forestier and Pierre-Yves Oudeyer · 2016
Later among the works it cites.
Early visual concept learning with unsupervised deep learning
Original
Irina Higgins, Loic Matthey, Xavier Glorot, Arka Pal, Benigno Uria, Charles Blundell, Shakir Mohamed, and Alexander Lerchner · 2016
Later among the works it cites.
Vime: Variational information maximizing exploration
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel · 2016
Later among the works it cites.
Intrinsic motivation, curiosity, and learning: Theory and applications in educational technologies
P-Y Oudeyer, Jacqueline Gottlieb, and Manuel Lopes · 2016
Later among the works it cites.
How evolution may work through curiosity-driven developmental process
Pierre-Yves Oudeyer and Linda B. Smith · 2016
Later among the works it cites.
Ladder variational autoencoders
Original
Casper K. Sonderby, Tapani Raiko, Lars Maaløe, Soren K. Sonderby, and Ole Winther · 2016
Later among the works it cites.
Improving variational auto-encoders using householder flow
Original
Jakub M. Tomczak and Max Welling · 2016
Later among the works it cites.
The intentional unintentional agent: Learning to solve many continuous control tasks simultaneously
Original
Serkan Cabi, Sergio G. Colmenarejo, Matthew W. Hoffman, Misha Denil, Ziyu Wang, and Nando De Freitas · 2017
Later among the works it cites.
Reverse curriculum generation for reinforcement learning
Original
Carlos Florensa, David Held, Markus Wulfmeier, and Pieter Abbeel · 2017
Later among the works it cites.
Intrinsically motivated goal exploration processes with automatic curriculum learning
Original
Sébastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer · 2017
Later among the works it cites.
Active inference, curiosity and insight
Karl J. Friston, Marco Lin, Christopher D. Frith, Giovanni Pezzulo, J. Allan Hobson, and Sasha Ondobaka · 2017
Later among the works it cites.
A predictive coding framework for a developmental agent: Speech motor skill acquisition and speech production
Shamima Najnin and Bonny Banerjee · 2017
Later among the works it cites.
Curiosity-driven exploration by self-supervised prediction
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell · 2017
Later among the works it cites.
# exploration: A study of count-based exploration for deep reinforcement learning
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel · 2017
Later among the works it cites.
Independently controllable factors
Original
Valentin Thomas, Jules Pondard, Emmanuel Bengio, Marc Sarfati, Philippe Beaudoin, Marie-Jean Meurs, Joelle Pineau, Doina Precup, and Yoshua Bengio · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…