2020

A Game Theoretic Framework for Model Based Reinforcement Learning

Rajeswaran, Aravind, Mordatch, Igor, Kumar, Vikash

Understand

Model-based reinforcement learning (MBRL) has recently gained immense interest due to its potential for sample efficiency and ability to incorporate off-policy data.

However, designing stable and efficient MBRL algorithms using rich function approximators have remained challenging.
To help expose the practical challenges in MBRL and simplify algorithm design from the lens of abstraction, we develop a new framework that casts MBRL as a game between: (1) a policy player, which attempts to maximize rewards under the learned model; (2) a model player, which attempts to fit the real-world data collected by the policy player.
For algorithm development, we construct a Stackelberg game between the two players, and show that it can be solved with approximate bi-level optimization.

Built on

On the optimality of sparse model-based planning for markov decision processes
Original
Agarwal, A., Kakade, S. M., and Yang, L. F · 1906
Earlier work this paper cites.
Benchmarking model-based reinforcement learning
Original
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., and Ba, J · 1907
Earlier work this paper cites.
Optimality and approximation with policy gradient methods in markov decision processes
Original
Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G · 1908
Earlier work this paper cites.
On solving minimax optimization locally: A follow-the-ridge approach
Original
Wang, Y., Zhang, G., and Ba, J · 1910
Earlier work this paper cites.
Market structure and equilibrium
von Stackelberg, H · 1934
Earlier work this paper cites.
System identification: Theory for the user
Ljung, L · 1987
Earlier work this paper cites.
Persistent excitation in adaptive systems
Narendra, K. S. and Annaswamy, A. M · 1987
Earlier work this paper cites.
Integrated architectures for learning, planning, and reacting based on approximating dynamic programming
Sutton, R. S · 1990
Earlier work this paper cites.
Markov decision processes: Discrete stochastic dynamic programming
Puterman, M. L · 1994
Earlier work this paper cites.
Robust and Optimal Control
Zhou, K., Doyle, J. C., and Glover, K · 1996
Earlier work this paper cites.
Actor-critic - type learning algorithms for markov decision processes
Konda, V. and Borkar, V. S · 1999
Earlier work this paper cites.
R-max - a general polynomial time algorithm for near-optimal reinforcement learning
Brafman, R. I. and Tennenholtz, M · 2001
Earlier work this paper cites.
Variance reduction techniques for gradient estimates in reinforcement learning
Greensmith, E., Bartlett, P. L., and Baxter, J · 2001
Earlier work this paper cites.
Approximately optimal approximate reinforcement learning
Kakade, S. M. and Langford, J · 2002
Earlier work this paper cites.
The implicit function theorem: History, theory, and applications
Krantz, S. G. and Parks, H. R · 2002
Earlier work this paper cites.
Exploration in metric state spaces
Kakade, S. M., Kearns, M., and Langford, J · 2003
Earlier work this paper cites.
Feedback systems an introduction for scientists and engineers
Åström, K. J. and Murray, R. M · 2004
Earlier work this paper cites.
Convergence rate of linear two-time-scale stochastic approximation
Konda, V. R. and Tsitsiklis, J. N · 2004
Earlier work this paper cites.
Exploration and apprenticeship learning in reinforcement learning
Abbeel, P. and Ng, A. Y · 2005
Earlier work this paper cites.
A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems
Todorov, E. and Li, W · 2005
Earlier work this paper cites.
Analysis of representations for domain adaptation
Ben-David, S., Blitzer, J., Crammer, K., and Pereira, F. C · 2006
Earlier work this paper cites.
Prediction, learning, and games
Cesa-Bianchi, N. and Lugosi, G · 2006
Earlier work this paper cites.
An overview of bilevel optimization
Colson, B., Marcotte, P., and Savard, G · 2007
Earlier work this paper cites.

Similar

Finite-time bounds for fitted value iteration
Munos, R. and Szepesvári, C · 2008
Cited alongside, same era.
Pilco: A model-based and data-efficient approach to policy search
Deisenroth, M. P. and Rasmussen, C. E · 2011
Cited alongside, same era.
Follow-the-regularized-leader and mirror descent: Equivalence theorems and l1 regularization
McMahan, H. B · 2011
Cited alongside, same era.
Agnostic system identification for model-based reinforcement learning
Ross, S. and Bagnell, J. A · 2012
Cited alongside, same era.
Online learning and online convex optimization
Shalev-Shwartz, S · 2012
Cited alongside, same era.
Mujoco: A physics engine for model-based control
Todorov, E., Erez, T., and Tassa, Y · 2012
Cited alongside, same era.
Learning neural network policies with guided policy search under unknown dynamics
Levine, S. and Abbeel, P · 2014
Cited alongside, same era.
Combining the benefits of function approximation and trajectory optimization
Mordatch, I. and Todorov, E · 2014
Cited alongside, same era.
Scheduled sampling for sequence prediction with recurrent neural networks
Original
Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N · 2015
Cited alongside, same era.
A comprehensive survey on safe reinforcement learning
Garcia, J. and Fernández, F · 2015
Cited alongside, same era.
Doubly robust off-policy value evaluation for reinforcement learning
Jiang, N. and Li, L · 2015
Cited alongside, same era.
Two time-scale stochastic approximation with controlled markov noise and off-policy temporal-difference learning
Karmakar, P. and Bhatnagar, S · 2015
Cited alongside, same era.

Then

Towards Generalization and Simplicity in Continuous Control
Rajeswaran, A., Lowrey, K., Todorov, E., and Kakade, S · 2017
Later among the works it cites.
Information theoretic mpc for model-based reinforcement learning
Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., and Theodorou, E · 2017
Later among the works it cites.
Sample-efficient reinforcement learning with stochastic ensemble value expansion
Original
Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H · 2018
Later among the works it cites.
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, K., Calandra, R., McAllister, R., and Levine, S · 2018
Later among the works it cites.
More robust doubly robust off-policy evaluation
Farajtabar, M., Chow, Y., and Ghavamzadeh, M · 2018
Later among the works it cites.
Model-based value estimation for efficient model-free reinforcement learning
Original
Feinberg, V., Wan, A., Stoica, I., Jordan, M. I., Gonzalez, J., and Levine, S · 2018
Later among the works it cites.
Soft actor-critic algorithms and applications
Original
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S · 2018
Later among the works it cites.
Provably efficient maximum entropy exploration
Hazan, E., Kakade, S. M., Singh, K., and Soest, A. V · 2018
Later among the works it cites.
Model-ensemble trust-region policy optimization
Original
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., and Abbeel, P · 2018
Later among the works it cites.
On first-order meta-learning algorithms
Original
Nichol, A., Achiam, J., and Schulman, J · 2018
Later among the works it cites.
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S · 2018
Later among the works it cites.
Variance reduction for policy gradient with action-dependent factorized baselines
Wu, C., Rajeswaran, A., Duan, Y., Kumar, V., Bayen, A. M., Kakade, S. M., Mordatch, I., and Abbeel, P · 2018
Later among the works it cites.
Algorithmic framework for model-based reinforcement learning with theoretical guarantees
Original
Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T · 2018
Later among the works it cites.
Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost
Zhu, H., Gupta, A., Rajeswaran, A., Levine, S., and Kumar, V · 2018
Later among the works it cites.
ROBEL: RObotics BEnchmarks for Learning with low-cost robots
Ahn, M., Zhu, H., Hartikainen, K., Ponte, H., Gupta, A., Levine, S., and Kumar, V · 2019
Later among the works it cites.
Convergence of learning dynamics in stackelberg games
Original
Fiez, T., Chasnov, B., and Ratliff, L. J · 2019
Later among the works it cites.
When to trust your model: Model-based policy optimization
Original
Janner, M., Fu, J., Zhang, M., and Levine, S · 2019
Later among the works it cites.
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
Lowrey, K., Rajeswaran, A., Kakade, S., Todorov, E., and Mordatch, I · 2019
Later among the works it cites.
Deep dynamics models for learning dexterous manipulation
Original
Nagabandi, A., Konoglie, K., Levine, S., and Kumar, V · 2019
Later among the works it cites.
Self-supervised exploration via disagreement
Original
Pathak, D., Gandhi, D., and Gupta, A · 2019
Later among the works it cites.
Meta-learning with implicit gradients
Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S · 2019
Later among the works it cites.
Competitive gradient descent
Schäfer, F. and Anandkumar, A · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…