2019

Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models

Byravan, Arunkumar, Springenberg, Jost Tobias, Abdolmaleki, Abbas et al.

Understand

Humans are masters at quickly learning many complex tasks, relying on an approximate understanding of the dynamics of their environments.

In much the same way, we would like our learning agents to quickly adapt to new tasks.
In this paper, we explore how model-based Reinforcement Learning (RL) can facilitate transfer to new tasks.
We develop an algorithm that learns an action-conditional, predictive model of expected future observations, rewards and values from which a policy can be derived by following the gradient of the estimated value along imagined trajectories.

Built on

Integrated architectures for learning, planning, and reacting based on approximating dynamic programming
R. S. Sutton · 1990
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
R. J. Williams · 1992
Earlier work this paper cites.
The Cross Entropy Method: A Unified Approach To Combinatorial Optimization, Monte-carlo Simulation (Information Science and Statistics)
R. Y. Rubinstein and D. P. Kroese · 2004
Earlier work this paper cites.
Pilco: A model-based and data-efficient approach to policy search
M. Deisenroth and C. E. Rasmussen · 2011
Earlier work this paper cites.
A survey on policy search for robotics
M. P. Deisenroth, G. Neumann, J. Peters, et al · 2013
Earlier work this paper cites.
Auto-encoding variational bayes
D. P. Kingma and M. Welling · 2013
Earlier work this paper cites.
Stochastic backpropagation and approximate inference in deep generative models
Original
D. J. Rezende, S. Mohamed, and D. Wierstra · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
D. P. Kingma and J. Ba · 2014
Earlier work this paper cites.
Human-level control through deep reinforcement learning
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, et al · 2015
Earlier work this paper cites.
Continuous control with deep reinforcement learning
Original
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra · 2015
Earlier work this paper cites.
Learning continuous control policies by stochastic value gradients
N. Heess, G. Wayne, D. Silver, T. Lillicrap, T. Erez, and Y. Tassa · 2015
Earlier work this paper cites.
Embed to control: A locally linear latent dynamics model for control from raw images
M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller · 2015
Earlier work this paper cites.
From pixels to torques: Policy learning with deep dynamical models
Original
N. Wahlström, T. B. Schön, and M. P. Deisenroth · 2015
Earlier work this paper cites.
Fast and accurate deep network learning by exponential linear units (elus)
Original
D.-A. Clevert, T. Unterthiner, and S. Hochreiter · 2015
Earlier work this paper cites.
End-to-end training of deep visuomotor policies
S. Levine, C. Finn, T. Darrell, and P. Abbeel · 2016
Earlier work this paper cites.
Continuous deep q-learning with model-based acceleration
S. Gu, T. P. Lillicrap, I. Sutskever, and S. Levine · 2016
Earlier work this paper cites.
Wide residual networks
Original
S. Zagoruyko and N. Komodakis · 2016
Earlier work this paper cites.

Similar

Layer normalization
Original
J. L. Ba, J. R. Kiros, and G. E. Hinton · 2016
Cited alongside, same era.
Asynchronous methods for deep reinforcement learning
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Harley, T. P. Lillicrap, D. Silver, and K. Kavukcuoglu · 2016
Cited alongside, same era.
Emergence of locomotion behaviours in rich environments
Original
N. Heess, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, et al · 2017
Cited alongside, same era.
Imagination-augmented agents for deep reinforcement learning
Original
T. Weber, S. Racanière, D. P. Reichert, L. Buesing, A. Guez, D. J. Rezende, A. P. Badia, et al · 2017
Cited alongside, same era.
Proximal policy optimization algorithms
Original
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov · 2017
Cited alongside, same era.
Value prediction network
J. Oh, S. Singh, and H. Lee · 2017
Cited alongside, same era.
Stochastic neural networks for hierarchical reinforcement learning
C. Florensa, Y. Duan, and P. Abbeel · 2017
Cited alongside, same era.
Successor features for transfer in reinforcement learning
A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver · 2017
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
C. Finn, P. Abbeel, and S. Levine · 2017
Cited alongside, same era.
The devil is in the decoder
Original
Z. Wojna, V. Ferrari, S. Guadarrama, N. Silberman, L.-C. Chen, A. Fathi, and J. Uijlings · 2017
Cited alongside, same era.
Learning dexterous in-hand manipulation
Original
OpenAI · 2018
Cited alongside, same era.
A general reinforcement learning algorithm that masters chess, shogi, & go through self-play
D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, et al · 2018
Cited alongside, same era.

Then

Solar: Deep structured latent representations for model-based reinforcement learning
Original
M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine · 2018
Later among the works it cites.
Model-based value estimation for efficient model-free reinforcement learning
Original
V. Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine · 2018
Later among the works it cites.
Unsupervised predictive memory in a goal-directed agent
Original
G. Wayne, C.-C. Hung, D. Amos, M. Mirza, A. Ahuja, A. Grabska-Barwinska, et al · 2018
Later among the works it cites.
Learning by playing-solving sparse reward tasks from scratch
M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Van de Wiele, V. Mnih, et al · 2018
Later among the works it cites.
Sample-efficient reinforcement learning with stochastic ensemble value expansion
J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee · 2018
Later among the works it cites.
Combined reinforcement learning via abstract representations
Original
V. François-Lavet, Y. Bengio, D. Precup, and J. Pineau · 2018
Later among the works it cites.
Learning an embedding space for transferable robot skills
K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller · 2018
Later among the works it cites.
Model-based reinforcement learning via meta-policy optimization
I. Clavera, J. Rothfuss, J. Schulman, Y. Fujita, T. Asfour, and P. Abbeel · 2018
Later among the works it cites.
AlphaStar: Mastering the Real-Time Strategy Game StarCraft II, 2019
O. Vinyals, I. Babuschkin, J. Chung, M. Mathieu, M. Jaderberg, W. M. Czarnecki, et al · 2019
Closest in time.
Model-based reinforcement learning for atari
Original
L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, et al · 2019
Closest in time.
Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost
H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar · 2019
Closest in time.
Learning latent dynamics for planning from pixels
D. Hafner, T. P. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson · 2019
Closest in time.
Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration
Original
N. Watters, L. Matthey, M. Bosnjak, et al · 2019
Closest in time.
Deepmdp: Learning continuous latent space models for representation learning
C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare · 2019
Closest in time.
Exploiting hierarchy for learning and transfer in kl-regularized RL
Original
D. Tirumala, H. Noh, A. Galashov, L. Hasenclever, A. Ahuja, G. Wayne, et al · 2019
Closest in time.
Information asymmetry in KL-regularized RL
A. Galashov, S. Jayakumar, L. Hasenclever, D. Tirumala, J. Schwarz, G. Desjardins, W. M. Czarnecki, Y. W. Teh, et al · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…