2020

Objective Mismatch in Model-based Reinforcement Learning

Lambert, Nathan, Amos, Brandon, Yadan, Omry et al.

Understand

Model-based reinforcement learning (MBRL) has been shown to be a powerful framework for data-efficiently learning control of continuous tasks.

Recent work in MBRL has mostly focused on using more advanced function approximators and planning schemes, with little development of the general framework.
In this paper, we identify a fundamental issue of the standard MBRL framework -- what we call the objective mismatch issue.
Objective mismatch arises when one objective is optimized in the hope that a second, often uncorrelated, metric will also be optimized.

Built on

Dyna, an integrated architecture for learning, planning, and reacting
Richard S Sutton · 1991
Earlier work this paper cites.
Dynamic programming and optimal control , volume 1
Dimitri P Bertsekas · 1995
Earlier work this paper cites.
For model-based control design, closed-loop identification gives better performance
Håkan Hjalmarsson, Michel Gevers, and Franky De Bruyne · 1996
Earlier work this paper cites.
Robust and optimal control , volume 40
Kemin Zhou, John Comstock Doyle, Keith Glover, et al · 1996
Earlier work this paper cites.
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Marc P. Deisenroth and Carl E. Rasmussen · 2011
Earlier work this paper cites.
Optimal control theory: an introduction
Donald E Kirk · 2012
Earlier work this paper cites.
Intriguing properties of neural networks
Original
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus · 2013
Earlier work this paper cites.
The dependence of effective planning horizon on model accuracy
Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis · 2015
Earlier work this paper cites.
Embed to control: A locally linear latent dynamics model for control from raw images
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller · 2015
Earlier work this paper cites.
Goal-driven dynamics learning via Bayesian optimization
S. Bansal, Roberto Calandra, T. Xiao, S. Levine, and C. J. Tomlin · 2017
Earlier work this paper cites.

Similar

Value-aware loss function for model-based reinforcement learning
Amir-massoud Farahmand, Andre Barreto, and Daniel Nikovski · 2017
Cited alongside, same era.
Path integral networks: End-to-end differentiable optimal control
Original
Masashi Okada, Luca Rigazio, and Takenobu Aoshima · 2017
Cited alongside, same era.
Self-correcting models for model-based reinforcement learning
Erik Talvitie · 2017
Cited alongside, same era.
Information theoretic MPC for model-based reinforcement learning
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou · 2017
Cited alongside, same era.
Differentiable MPC for end-to-end planning and control
Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and J Zico Kolter · 2018
Cited alongside, same era.
Applied optimal control: optimization, estimation and control
Arthur Earl Bryson · 2018
Cited alongside, same era.
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine · 2018
Cited alongside, same era.
Iterative value-aware model learning
Amir-massoud Farahmand · 2018
Cited alongside, same era.
Learning actionable representations with goal-conditioned policies
Original
Dibya Ghosh, Abhishek Gupta, and Sergey Levine · 2018
Cited alongside, same era.
Learning latent dynamics for planning from pixels
Original
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson · 2018
Cited alongside, same era.

Then

Universal planning networks
Original
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn · 2018
Later among the works it cites.
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto · 2018
Later among the works it cites.
Deepmdp: Learning continuous latent space models for representation learning
Original
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare · 2019
Later among the works it cites.
When to trust your model: Model-based policy optimization
Original
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine · 2019
Later among the works it cites.
Learning dynamics model in reinforcement learning by incorporating the long term future
Original
Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, and Dhruv Batra · 2019
Later among the works it cites.
Robust reinforcement learning for continuous control with model misspecification
Original
Daniel J Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Timothy Mann, Todd Hester, and Martin Riedmiller · 2019
Later among the works it cites.
Learning stabilizable nonlinear dynamics with contraction-based regularization, 2019
Sumeet Singh, Spencer M. Richards, Vikas Sindhwani, Jean-Jacques E. Slotine, and Marco Pavone · 2019
Later among the works it cites.
Exploring model-based planning with policy networks
Original
Tingwu Wang and Jimmy Ba · 2019
Later among the works it cites.
Solar: Deep structured latent representations for model-based reinforcement learning
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…