2015

From Pixels to Torques: Policy Learning with Deep Dynamical Models

Wahlström, Niklas, Schön, Thomas B., Deisenroth, Marc Peter

Understand

Data-efficient learning in continuous state-action spaces using very high-dimensional observations remains a key challenge in developing fully autonomous systems.

In this paper, we consider one instance of this challenge, the pixels to torques problem, where an agent must learn a closed-loop control policy from pixel information only.
We introduce a data-efficient, model-based reinforcement learning algorithm that learns such a closed-loop policy directly from pixel information.
The key ingredient is a deep dynamical model that uses deep auto-encoders to learn a low-dimensional embedding of images jointly with a predictive model in this low-dimensional feature space.

Built on

Auto-association by multilayer perceptrons and singular value decomposition
Bourlard, Hervé and Kamp, Yves · 1988
Earlier work this paper cites.
An on-line algorithm for dynamic reinforcement learning and planning in reactive environments
Schmidhuber, Jürgen · 1990
Earlier work this paper cites.
Learning tasks from a single demonstration
Atkeson, Christopher G. and Schaal, S · 1997
Earlier work this paper cites.
Learning from demonstration
Schaal, Stefan · 1997
Earlier work this paper cites.
Exploiting model uncertainty estimates for safe dynamic control learning
Schneider, Jeff G · 1997
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y, Bottou, L, Bengio, Y, and Haffner, P · 1998
Earlier work this paper cites.
Reinforcement Learning: An Introduction
Sutton, Richard S. and Barto, Andrew G · 1998
Earlier work this paper cites.
System Identification: Theory for the User
Ljung, L · 1999
Earlier work this paper cites.
Autonomous helicopter control using reinforcement learning policy search methods
Bagnell, James A. and Schneider, Jeff G · 2001
Earlier work this paper cites.
Reducing the dimensionality of data with neural networks
Hinton, G and Salakhutdinov, R · 2006
Earlier work this paper cites.
Numerical Optimization
Nocedal, J. and Wright, S. J · 2006
Earlier work this paper cites.

Similar

Gaussian Processes for Machine Learning
Rasmussen, Carl E. and Williams, Christopher K. I · 2006
Cited alongside, same era.
Greedy layer-wise training of deep networks
Bengio, Yoshua, Lamblin, Pascal, Popovici, Dan, and Larochelle, Hugo · 2007
Cited alongside, same era.
A new neural networks based adaptive model predictive control for unknown multiple variable non-linear systems
Sha, Daohang · 2008
Cited alongside, same era.
Extracting and composing robust features with denoising autoencoders
Vincent, P, Larochelle, H, Bengio, Y, and Manzagol, Pierre-Antoine · 2008
Cited alongside, same era.
Gaussian process dynamic programming
Deisenroth, Marc P., Rasmussen, Carl E., and Peters, Jan · 2009
Cited alongside, same era.
Berlin Summit on Robotics: Conference Report , chapter Is Robotics in Need of a Paradigm Shift?, pp. 1–10
Brock, Oliver · 2011
Cited alongside, same era.
Intrinsically motivated neuroevolution for vision-based reinforcement learning
Cuccu, Giuseppe, Luciw, Matthew, Schmidhuber, Jürgen, and Gomez, Faustino · 2011
Cited alongside, same era.
Hierarchical, heterogeneous control using reinforcement learning
Abramova, Ekatarina, Dickens, Luke, Kuhn, Daniel, and Faisal, A. Aldo · 2012
Cited alongside, same era.
Autonomous reinforcement learning on raw visual input data in a real-world application
Lange, Sascha, Riedmiller, Martin, and Voigtländer, Arne · 2012
Cited alongside, same era.
Learning states representations in POMDP
Original
Contardo, Gabriella, Denoyer, Ludovic, Artieres, Thierry, and Gallinari, Patrick · 2013
Cited alongside, same era.

Then

Evolving large-scale neural networks for vision-based reinforcement learning
Koutnik, Jan, Cuccu, Giuseppe, Schmidhuber, Jürgen, and Gomez, Faustino · 2013
Later among the works it cites.
Approximate real-time optimal control based on sparse Gaussian process models
Boedecker, Joschka, Springenberg, Jost Tobias, Wülfing, Jan, and Riedmiller, Martin · 2014
Later among the works it cites.
Learning predictive models of a depth camera & manipulator from raw execution traces
Boots, Byron, Byravan, Arunkumar, and Fox, Dieter · 2014
Later among the works it cites.
Model predictive control: Recent developments and future promise
Mayne, David Q · 2014
Later among the works it cites.
Probabilistic differential dynamic programming
Pan, Yunpeng and Theodorou, Evangelos · 2014
Later among the works it cites.
Gaussian processes for data-efficient learning in robotics and control
Deisenroth, Marc P., Fox, Dieter, and Rasmussen, Carl E · 2015
Closest in time.
End-to-end training of deep visuomotor policies
Original
Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel, Pieter · 2015
Closest in time.
Human-level control through deep reinforcement learning
Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, and et al · 2015
Closest in time.
Learning of non-parametric control policies with high-dimensional state features
van Hoof, Herke, Peters, Jan, and Neumann, Gerhard · 2015
Closest in time.
Learning deep dynamical models from image pixels
Wahlström, Niklas, Schön, Thomas B., and Deisenroth, Marc P · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…