2018

Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Lowrey, Kendall, Rajeswaran, Aravind, Kakade, Sham et al.

Understand

We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world.

Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration.
We study how local trajectory optimization can cope with approximation errors in the value function, and can stabilize and accelerate value function learning.
Conversely, we also study how approximate value functions can help reduce the planning horizon and allow for better policies beyond local solutions.

Built on

Differential Dynamic Programming
David Jacobson and David Mayne · 1970
Earlier work this paper cites.
System identification: theory for the user
Lennart Ljung · 1987
Earlier work this paper cites.
Model predictive control: Theory and practice - a survey
Carlos E. Garcia, David M. Prett, and Manfred Morari · 1989
Earlier work this paper cites.
Using local trajectory optimizers to speed up global optimization in dynamic programming
Christopher G. Atkeson · 1993
Earlier work this paper cites.
Learning to act using real-time dynamic programming
Andrew G. Barto, Steven J. Bradtke, and Satinder P. Singh · 1995
Earlier work this paper cites.
Neuro-dynamic Programming
Dimitri Bertsekas and John Tsitsiklis · 1996
Earlier work this paper cites.
Policy invariance under reward transformations: Theory and application to reward shaping
Andrew Y. Ng, Daishi Harada, and Stuart J. Russell · 1999
Earlier work this paper cites.
Finite-time analysis of the multiarmed bandit problem
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer · 2002
Earlier work this paper cites.
Near-optimal reinforcement learning in polynomial time
Michael Kearns and Satinder Singh · 2002
Earlier work this paper cites.
A survey of industrial model predictive control technology
S. Joe Qina and Thomas A. Badgwellb · 2003
Earlier work this paper cites.
Policy search by dynamic programming
J. Andrew Bagnell, Sham M. Kakade, Andrew Y. Ng, and Jeff G. Schneider · 2003
Earlier work this paper cites.
Feedback systems an introduction for scientists and engineers
Karl Johan Åström and Richard M. Murray · 2004
Earlier work this paper cites.
A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems
Emanuel Todorov and Weiwei Li · 2005
Earlier work this paper cites.
Intrinsically motivated reinforcement learning
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh · 2005
Earlier work this paper cites.
Finite-time bounds for fitted value iteration
Rémi Munos and Csaba Szepesvári · 2008
Earlier work this paper cites.
Compositionality of optimal control laws
Emanuel Todorov · 2009
Earlier work this paper cites.
A contextual-bandit approach to personalized news article recommendation
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire · 2010
Earlier work this paper cites.
A generalized path integral control approach to reinforcement learning
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal · 2010
Earlier work this paper cites.
A reduction of imitation learning and structured prediction to no-regret online learning
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell · 2011
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
Emanuel Todorov, Tom Erez, and Yuval Tassa · 2012
Earlier work this paper cites.

Similar

Agnostic system identification for model-based reinforcement learning
Stéphane Ross and J. Andrew Bagnell · 2012
Cited alongside, same era.
Synthesis and stabilization of complex behaviors through online trajectory optimization
Yuval Tassa, Tom Erez, and Emanuel Todorov · 2012
Cited alongside, same era.
Value function approximation and model predictive control
Mingyuan Zhong, Mikala Johnson, Yuval Tassa, Tom Erez, and Emanuel Todorov · 2013
Cited alongside, same era.
Guided policy search
Sergey Levine and Vladlen Koltun · 2013
Cited alongside, same era.
(more) efficient reinforcement learning via posterior sampling
Ian Osband, Daniel Russo, and Benjamin Van Roy · 2013
Cited alongside, same era.
Control-limited differential dynamic programming
Yuval Tassa, Nicolas Mansard, and Emanuel Todorov · 2014
Cited alongside, same era.
Universal convexification via risk-aversion
Krishnamurthy Dvijotham, Maryam Fazel, and Emanuel Todorov · 2014
Cited alongside, same era.
Combining the benefits of function approximation and trajectory optimization
Igor Mordatch and Emanuel Todorov · 2014
Cited alongside, same era.
High-dimensional continuous control using generalized advantage estimation
Original
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al · 2015
Cited alongside, same era.
Incentivizing exploration in reinforcement learning with deep predictive models
Original
Bradly C Stadie, Sergey Levine, and Pieter Abbeel · 2015
Cited alongside, same era.
Epopt: Learning robust neural network policies using model ensembles
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine · 2016
Cited alongside, same era.

Then

Sim-to-real transfer of robotic control with dynamics randomization
Original
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel · 2017
Later among the works it cites.
Mastering the game of go without human knowledge
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al · 2017
Later among the works it cites.
Thinking fast and slow with deep learning and tree search
Thomas Anthony, Zheng Tian, and David Barber · 2017
Later among the works it cites.
Augmenting sampling based controllers with machine learning
Joose Rajamäki and Perttu Hämäläinen · 2017
Later among the works it cites.
Towards Generalization and Simplicity in Continuous Control
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham Kakade · 2017
Later among the works it cites.
Parameter space noise for exploration
Original
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz · 2017
Later among the works it cites.
Noisy networks for exploration
Original
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al · 2017
Later among the works it cites.
Curiosity-driven exploration by self-supervised prediction
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell · 2017
Later among the works it cites.
Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system
Original
Kendall Lowrey, Svetoslav Kolev, Jeremy Dao, Aravind Rajeswaran, and Emanuel Todorov · 2018
Closest in time.
Learning dexterous in-hand manipulation
OpenAI · 2018
Closest in time.
Randomized prior functions for deep reinforcement learning
Original
Ian Osband, John Aslanides, and Albin Cassirer · 2018
Closest in time.
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine · 2018
Closest in time.
Divide-and-conquer reinforcement learning
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine · 2018
Closest in time.
Dual policy iteration
Original
Wen Sun, Geoffrey J. Gordon, Byron Boots, and J. Andrew Bagnell · 2018
Closest in time.
Universal planning networks
Original
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn · 2018
Closest in time.
Model-based value estimation for efficient model-free reinforcement learning
Original
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph Gonzalez, and Sergey Levine · 2018
Closest in time.
Sample-efficient reinforcement learning with stochastic ensemble value expansion
Original
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee · 2018
Closest in time.
Sample-efficient deep rl with generative adversarial tree search
Original
Kamyar Azizzadenesheli, Brandon Yang, Weitang Liu, Emma Brunskill, Zachary Chase Lipton, and Anima Anandkumar · 2018
Closest in time.
Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine · 2018
Closest in time.
Truncated horizon policy search: Combining reinforcement learning & imitation learning
Original
Wen Sun, J. Andrew Bagnell, and Byron Boots · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…