2020

MOReL : Model-Based Offline Reinforcement Learning

Kidambi, Rahul, Rajeswaran, Aravind, Netrapalli, Praneeth et al.

Understand

In offline reinforcement learning (RL), the goal is to learn a highly rewarding policy based solely on a dataset of historical interactions with the environment.

The ability to train RL policies offline can greatly expand the applicability of RL, its data efficiency, and its experimental velocity.
Prior work in offline RL has been confined almost exclusively to model-free RL approaches.
In this work, we present MOReL, an algorithmic framework for model-based offline RL.

Built on

Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections
Original
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li · 1906
Earlier work this paper cites.
On the optimality of sparse model-based planning for markov decision processes
Original
Alekh Agarwal, Sham M. Kakade, and Lin F. Yang · 1906
Earlier work this paper cites.
Striving for simplicity in off-policy deep reinforcement learning
Original
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi · 1907
Earlier work this paper cites.
Algaedice: Policy gradient from arbitrary experience
Original
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans · 1912
Earlier work this paper cites.
The DARPA speech recognition research database: Specification and status
W. Fisher, G. Doddington, and K. Goudie-Marshall · 1986
Earlier work this paper cites.
Learning from delayed rewards
Chris Watkins · 1989
Earlier work this paper cites.
Neuro-Dynamic Programming
Dimitri P. Bertsekas and John N. Tsitsiklis · 1996
Earlier work this paper cites.
Reinforcement Learning: An Introduction
R. S. Sutton and A. G. Barto · 1998
Earlier work this paper cites.
Rapidly-exploring random trees: A new tool for path planning, 1998
Steven M. Lavalle · 1998
Earlier work this paper cites.
R-max - a general polynomial time algorithm for near-optimal reinforcement learning
Ronen I. Brafman and Moshe Tennenholtz · 2001
Earlier work this paper cites.
A natural policy gradient
Sham M. Kakade · 2001
Earlier work this paper cites.
Near optimal reinforcement learning in polynomial time
Michael Kearns and Satinder Singh · 2002
Earlier work this paper cites.
Approximately optimal approximate reinforcement learning
Sham M. Kakade and John Langford · 2002
Earlier work this paper cites.
Exploration in metric state spaces
Sham M. Kakade, Michael J. Kearns, and John Langford · 2003
Earlier work this paper cites.
A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems
Emanuel Todorov and Weiwei Li · 2005
Earlier work this paper cites.
Finite-time bounds for fitted value iteration
Rémi Munos and Csaba Szepesvari · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L. J. Li, K.Li, and L. Fei Fei · 2009
Earlier work this paper cites.
Learning from logged implicit exploration data
Original
Alexander L. Strehl, John Langford, and Sham M. Kakade · 2010
Earlier work this paper cites.
Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms, 2010
Lihong Li, Wei Chu, John Langford, and Xuanhui Wang · 2011
Earlier work this paper cites.
Deep neural networks for acoustic modeling in speech recognition, November 26 2012
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel rahman Mohamed, Navdeep Jaitly, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury · 2012
Earlier work this paper cites.
Batch reinforcement learning
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller · 2012
Earlier work this paper cites.
Synthesis and stabilization of complex behaviors through online trajectory optimization
Yuval Tassa, Tom Erez, and Emanuel Todorov · 2012
Earlier work this paper cites.
A survey of monte carlo tree search methods
Cameron Browne, Edward Jack Powley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez Liebana, Spyridon Samothrakis, and Simon Colton · 2012
Earlier work this paper cites.
Agnostic system identification for model-based reinforcement learning
Stephane Ross and Drew Bagnell · 2012
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
Emanuel Todorov, Tom Erez, and Yuval Tassa · 2012
Earlier work this paper cites.
One billion word benchmark for measuring progress in statistical language modeling
Original
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn · 2013
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Original
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean · 2013
Earlier work this paper cites.
Safe reinforcement learning
Philip S Thomas · 2014
Earlier work this paper cites.
Batch learning from logged bandit feedback through counterfactual risk minimization
Adith Swaminathan and Thorsten Joachims · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis · 2015
Earlier work this paper cites.

Similar

Improving multi-step prediction of learned time series models
Arun Venkatraman, Martial Hebert, and J. Andrew Bagnell · 2015
Cited alongside, same era.
Scheduled sampling for sequence prediction with recurrent neural networks
Original
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer · 2015
Cited alongside, same era.
Adam: A method for stochastic optimization
Diederik P. Kingma and Jimmy Ba · 2015
Cited alongside, same era.
Trust region policy optimization
Original
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel · 2015
Cited alongside, same era.
Deep neural networks for youtube recommendations
Paul Covington, Jay Adams, and Emre Sargin · 2016
Cited alongside, same era.
Continuous control with deep reinforcement learning
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra · 2016
Cited alongside, same era.
Safe policy improvement by minimizing robust baseline regret
Mohammad Ghavamzadeh, Marek Petrik, and Yinlam Chow · 2016
Cited alongside, same era.
Openai gym, 2016
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton · 2017
Cited alongside, same era.
Safe policy improvement with baseline bootstrapping
Original
Romain Laroche and Paul Trichelair · 2017
Cited alongside, same era.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Cited alongside, same era.
End-to-end offline goal-oriented dialog policy learning via policy gradient
Original
Li Zhou, Kevin Small, Oleg Rokhlenko, and Charles Elkan · 2017
Cited alongside, same era.

Then

Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine · 2018
Later among the works it cites.
Grandmaster level in starcraft ii using multi-agent reinforcement learning
Oriol Vinyals, Igor Babuschkin, Wojciech Marian Czarnecki, Michaël Mathieu, Andrew Joseph Dudzik, Junyoung Chung, Duck Hwan Choi, Richard W. Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver · 2019
Later among the works it cites.
Preventing undesirable behavior of intelligent machines
Philip S. Thomas, Bruno Castro da Silva, Andrew G. Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill · 2019
Later among the works it cites.
Stabilizing off-policy q-learning via bootstrapping error reduction
Original
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine · 2019
Later among the works it cites.
Behavior regularized offline reinforcement learning
Original
Yifan Wu, George Tucker, and Ofir Nachum · 2019
Later among the works it cites.
Way off-policy batch deep reinforcement learning of implicit human preferences in dialog
Original
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard · 2019
Later among the works it cites.
Tossingbot: Learning to throw arbitrary objects with residual physics
Original
Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodríguez, and Thomas A. Funkhouser · 2019
Later among the works it cites.
When to trust your model: Model-based policy optimization
Original
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine · 2019
Later among the works it cites.
Deep inverse reinforcement learning for sepsis treatment
Chao Yu, Guoqi Ren, and Jiming Liu 0001 · 2019
Later among the works it cites.
Lessons from real-world reinforcement learning in a customer support bot
Original
Nikos Karampatziakis, Sebastian Kochman, Jade Huang, Paul Mineiro, Kathy Osborne, and Weizhu Chen · 2019
Later among the works it cites.
Off-policy policy gradient with state distribution correction
Original
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill · 2019
Later among the works it cites.
Off-policy deep reinforcement learning by bootstrapping the covariate shift
Carles Gelada and Marc G. Bellemare · 2019
Later among the works it cites.
Information-theoretic considerations in batch reinforcement learning
Jinglin Chen and Nan Jiang · 2019
Later among the works it cites.
Mo’ states mo’ problems: Emergency stop mechanisms from observation
Original
Samuel K. Ainsworth, Matt Barnes, and Siddhartha S. Srinivasa · 2019
Later among the works it cites.
P3o: Policy-on policy-off policy optimization
Original
Rasool Fakoor, Pratik Chaudhari, and Alexander J. Smola · 2019
Later among the works it cites.
Exploration by random network distillation
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov · 2019
Later among the works it cites.
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch · 2019
Later among the works it cites.
Benchmarking model-based reinforcement learning
Original
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, S. Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba · 2019
Later among the works it cites.
Deep dynamics models for learning dexterous manipulation
Original
Anusha Nagabandi, Kurt Konoglie, Sergey Levine, and Vikash Kumar · 2019
Later among the works it cites.
Data efficient reinforcement learning for legged robots
Original
Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Tingnan Zhang, Jie Tan, and Vikas Sindhwani · 2019
Later among the works it cites.
Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma · 2019
Later among the works it cites.
A Game Theoretic Framework for Model-Based Reinforcement Learning
Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar · 2020
Closest in time.
Mopo: Model-based offline policy optimization
Original
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma · 2020
Closest in time.
Planning and execution using inaccurate models with provable guarantees, 2020
Anirudh Vemula, Yash Oza, J. Andrew Bagnell, and Maxim Likhachev · 2020
Closest in time.
D4rl: Datasets for deep data-driven reinforcement learning
Original
Justin Fu, Aviral Kumar, Ofir Nachum, G. Tucker, and S. Levine · 2020
Closest in time.
Conservative q-learning for offline reinforcement learning
Original
Aviral Kumar, Aurick Zhou, G. Tucker, and S. Levine · 2020
Closest in time.
Offline reinforcement learning: Tutorial, review, and perspectives on open problems
Original
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu · 2020
Closest in time.
Reinforcement learning via fenchel-rockafellar duality
Original
Ofir Nachum and Bo Dai · 2020
Closest in time.
Exploring model-based planning with policy networks
Original
Tingwu Wang and Jimmy Ba · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…