2020

Ready Policy One: World Building Through Active Learning

Ball, Philip, Parker-Holder, Jack, Pacchiano, Aldo et al.

Understand

Model-Based Reinforcement Learning (MBRL) offers a promising direction for sample efficient learning, often achieving state of the art results for continuous control tasks.

However, many existing MBRL methods rely on combining greedy policies with exploration heuristics, and even those which utilize principled exploration bonuses construct dual objectives in an ad hoc fashion.
In this paper we introduce Ready Policy One (RP1), a framework that views MBRL as an active learning problem, where we aim to improve the world model in the fewest samples possible.
RP1 achieves this by utilizing a hybrid objective function, which crucially adapts during optimization, allowing the algorithm to trade off reward v.s.

Built on

A Markovian decision process
R. Bellman · 1957
Earlier work this paper cites.
Integrated architectures for learning, planning, and reacting based on approximating dynamic programming
R. S. Sutton · 1990
Earlier work this paper cites.
Curious model-building control systems
J. Schmidhuber · 1991
Earlier work this paper cites.
Dyna, an integrated architecture for learning, planning, and reacting
R. S. Sutton · 1991
Earlier work this paper cites.
Information-based objective functions for active data selection
D. J. C. MacKay · 1992
Earlier work this paper cites.
Efficient global optimization of expensive black-box functions
D. R. Jones, M. Schonlau, and W. J. Welch · 1998
Earlier work this paper cites.
Information Theory, Inference & Learning Algorithms
D. J. C. MacKay · 2002
Earlier work this paper cites.
R-max - a general polynomial time algorithm for near-optimal reinforcement learning
R. I. Brafman and M. Tennenholtz · 2003
Earlier work this paper cites.
Intrinsically motivated reinforcement learning
N. Chentanez, A. G. Barto, and S. P. Singh · 2005
Earlier work this paper cites.
Balancing exploration and exploitation: A new algorithm for active machine learning
T. Osugi, D. Kun, and S. Scott · 2005
Earlier work this paper cites.
Prediction, learning, and games
N. Cesa-Bianchi and G. Lugosi · 2006
Earlier work this paper cites.
Efficient exploration through active learning for value function approximation in reinforcement learning
T. Akiyama, H. Hachiya, and M. Sugiyama · 2010
Earlier work this paper cites.
Formal theory of creativity, fun, and intrinsic motivation (1990–2010)
J. Schmidhuber · 2010
Earlier work this paper cites.
Dynamic batch mode active learning
S. Chakraborty, V. Balasubramanian, and S. Panchanathan · 2011
Earlier work this paper cites.
Pilco: A model-based and data-efficient approach to policy search
M. P. Deisenroth and C. E. Rasmussen · 2011
Earlier work this paper cites.
Sequential model-based optimization for general algorithm configuration
F. Hutter, H. H. Hoos, and K. Leyton-Brown · 2011
Earlier work this paper cites.
Planning to be surprised: Optimal Bayesian exploration in dynamic environments
Y. Sun, F. Gomez, and J. Schmidhuber · 2011
Earlier work this paper cites.

Similar

Exploration in model-based reinforcement learning by empirically estimating learning progress
M. Lopes, T. Lang, M. Toussaint, and P. yves Oudeyer · 2012
Cited alongside, same era.
Active subspace methods in theory and practice: Applications to kriging surfaces
P. G. Constantine, E. Dow, and Q. Wang · 2014
Cited alongside, same era.
Variational information maximisation for intrinsically motivated reinforcement learning
S. Mohamed and D. J. Rezende · 2015
Cited alongside, same era.
Trust region policy optimization
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz · 2015
Cited alongside, same era.
OpenAI Gym, 2016
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba · 2016
Cited alongside, same era.
Benchmarking deep reinforcement learning for continuous control
Original
Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel · 2016
Cited alongside, same era.
Vime: Variational information maximizing exploration
R. Houthooft, X. Chen, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel · 2016
Cited alongside, same era.
Deep reinforcement learning that matters
Original
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger · 2017
Cited alongside, same era.
Count-based exploration with neural density models
G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos · 2017
Cited alongside, same era.
Curiosity-driven exploration by self-supervised prediction
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell · 2017
Cited alongside, same era.
Proximal policy optimization algorithms, 2017
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov · 2017
Cited alongside, same era.
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
K. Chua, R. Calandra, R. McAllister, and S. Levine · 2018
Cited alongside, same era.

Then

Soft actor-critic algorithms and applications
Original
T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine · 2018
Later among the works it cites.
Are deep policy gradient algorithms truly policy gradient algorithms?
Original
A. Ilyas, L. Engstrom, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry · 2018
Later among the works it cites.
Model-ensemble trust-region policy optimization
T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel · 2018
Later among the works it cites.
Large-scale study of curiosity-driven learning
Y. Burda, H. Edwards, D. Pathak, A. J. Storkey, T. Darrell, and A. A. Efros · 2019
Later among the works it cites.
If maxent rl is the answer, what is the question?, 2019
B. Eysenbach and S. Levine · 2019
Later among the works it cites.
Learning to predict without looking ahead: World models without forward prediction
D. Freeman, D. Ha, and L. Metz · 2019
Later among the works it cites.
Learning latent dynamics for planning from pixels
D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson · 2019
Later among the works it cites.
Explicit explore-exploit algorithms in continuous state spaces
M. Henaff · 2019
Later among the works it cites.
When to trust your model: Model-based policy optimization
Original
M. Janner, J. Fu, M. Zhang, and S. Levine · 2019
Later among the works it cites.
Deep dynamics models for learning dexterous manipulation, 2019
A. Nagabandi, K. Konoglie, S. Levine, and V. Kumar · 2019
Later among the works it cites.
Self-supervised exploration via disagreement
D. Pathak, D. Gandhi, and A. Gupta · 2019
Later among the works it cites.
Active inference: demystified and compared
Original
N. Sajid, P. J. Ball, and K. J. Friston · 2019
Later among the works it cites.
Mastering Atari, Go, chess and shogi by planning with a learned model, 2019
J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver · 2019
Later among the works it cites.
Model-based active exploration
P. Shyam, W. Jaskowski, and F. Gomez · 2019
Later among the works it cites.
Benchmarking model-based reinforcement learning
Original
T. Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois, S. Zhang, G. Zhang, P. Abbeel, and J. Ba · 2019
Later among the works it cites.
Dream to control: Learning behaviors by latent imagination
D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi · 2020
Closest in time.
Model based reinforcement learning for atari
Ł. Kaiser, M. Babaeizadeh, P. Miłos, B. Osiński, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, and H. Michalewski · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…