2019

Improving Generalization in Meta Reinforcement Learning using Learned Objectives

Kirsch, Louis, van Steenkiste, Sjoerd, Schmidhuber, Jürgen

Understand

Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans.

Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process.
MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that decides how future individuals will learn.
Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training.

Built on

Evolutionary principles in self-referential learning. Diploma thesis, Institut für Informatik, Technische Universität München, 1987
Jürgen Schmidhuber · 1987
Earlier work this paper cites.
On the Use of Backpropagation in Associative Reinforcement Learning
R J Williams · 1988
Earlier work this paper cites.
Learning a synaptic learning rule
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier · 1990
Earlier work this paper cites.
Making the world differentiable: On Using Fully Recurrent Self-Supervised Neural Networks for Dynamic Reinforcement Learning and Planning in Non-Stationary Environments
Jürgen Schmidhuber · 1990
Earlier work this paper cites.
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Ronald J Williams · 1992
Earlier work this paper cites.
Feudal Reinforcement Learning
P Dayan and G Hinton · 1993
Earlier work this paper cites.
A self-referential weight matrix
Jürgen Schmidhuber · 1993
Earlier work this paper cites.
On learning how to learn learning strategies
J Schmidhuber · 1994
Earlier work this paper cites.
HQ-Learning: Discovering Markovian Subgoals for Non-Markovian Reinforcement Learning
M Wiering and J Schmidhuber · 1996
Earlier work this paper cites.
Long Short-Term Memory
S Hochreiter and J Schmidhuber · 1997
Earlier work this paper cites.
Reduction of the Hodgkin-Huxley equations to a single-variable threshold model
Werner M Kistler, Wulfram Gerstner, and J Leo van Hemmen · 1997
Earlier work this paper cites.
Direct policy search and uncertain policy evaluation
J Schmidhuber and J Zhao · 1998
Earlier work this paper cites.
Learning to Forget: Continual Prediction with LSTM
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins · 2000
Earlier work this paper cites.
Learning to learn using gradient descent
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell · 2001
Earlier work this paper cites.
Natural Evolution Strategies
Daan Wierstra, Tom Schaul, Jan Peters, and Jürgen Schmidhuber · 2008
Earlier work this paper cites.
Intrinsically motivated reinforcement learning
Satinder Singh, Satinder Singh, A.G. Barto, A.G. Barto, Nuttapong Chentanez, and Nuttapong Chentanez · 2010
Earlier work this paper cites.
Evolution of reward functions for reinforcement learning
Scott Niekum, Lee Spector, and Andrew Barto · 2011
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
Emanuel Todorov, Tom Erez, and Yuval Tassa · 2012
Earlier work this paper cites.
Playing Atari with Deep Reinforcement Learning
Original
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller · 2013
Earlier work this paper cites.
Auto-Encoding Variational Bayes
Diederik P Kingma and Max Welling · 2014
Earlier work this paper cites.

Similar

Stochastic Backpropagation and Approximate Inference in Deep Generative Models
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra · 2014
Cited alongside, same era.
Deterministic policy gradient algorithms
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller · 2014
Cited alongside, same era.
Adam: A Method for Stochastic Optimization
Diederik P Kingma and Jimmy Ba · 2015
Cited alongside, same era.
Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
Original
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov · 2015
Cited alongside, same era.
Learning to learn by gradient descent by gradient descent
Marcin Andrychowicz, Misha Denil, Sergio Gómez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas · 2016
Cited alongside, same era.
Layer Normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Cited alongside, same era.
OpenAI Gym
Original
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Cited alongside, same era.
RLˆ2: Fast Reinforcement Learning via Slow Reinforcement Learning
Original
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel · 2016
Cited alongside, same era.
VIME: Variational Information Maximizing Exploration
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel · 2016
Cited alongside, same era.
Learning to Optimize
Original
Ke Li and Jitendra Malik · 2016
Cited alongside, same era.
Continuous control with deep reinforcement learning
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra · 2016
Cited alongside, same era.
Policy Distillation
Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell · 2016
Cited alongside, same era.

Then

Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
Chelsea Finn and Sergey Levine · 2018
Later among the works it cites.
Addressing function approximation error in actor-critic methods
Scott Fujimoto, Herke van Hoof, and David Meger · 2018
Later among the works it cites.
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths · 2018
Later among the works it cites.
Recurrent world models facilitate policy evolution
David Ha and Jürgen Schmidhuber · 2018
Later among the works it cites.
Evolved Policy Gradients
Rein Houthooft, Richard Y. Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel · 2018
Later among the works it cites.
Rllib: Abstractions for distributed reinforcement learning
Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg, Joseph E Gonzalez, Michael I Jordan, and Ion Stoica · 2018
Later among the works it cites.
A Simple Neural Attentive Meta-Learner
Nikhil Mishra, Mostafa Rohaninejad, and Xi UC Chen Pieter Abbeel Berkeley · 2018
Later among the works it cites.
Gotta Learn Fast: A New Benchmark for Generalization in RL
Original
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman Openai · 2018
Later among the works it cites.
Multi-goal reinforcement learning: Challenging robotics environments and request for research
Original
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, and others · 2018
Later among the works it cites.
Meta-gradient reinforcement learning
Zhongwen Xu, Hado Van Hasselt, and David Silver · 2018
Later among the works it cites.
Bayesian Model-Agnostic Meta-Learning
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn · 2018
Later among the works it cites.
One-shot imitation from observing humans via domain-adaptive meta-learning
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine · 2018
Later among the works it cites.
On learning intrinsic rewards for policy gradient methods
Zeyu Zheng, Junhyuk Oh, and Satinder Singh · 2018
Later among the works it cites.
Meta-learning via learned loss
Original
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier · 2019
Closest in time.
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
Original
Jeff Clune · 2019
Closest in time.
Human-level performance in 3D multiplayer games with population-based reinforcement learning
Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel · 2019
Closest in time.
Learning Unsupervised Learning Rules
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein · 2019
Closest in time.
Meta-Learning with Latent Embedding Optimization
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell · 2019
Closest in time.
Meta-learning curiosity algorithms
Ferran Alet, Martin F Schneider, Tomas Lozano-Perez, and Leslie Pack Kaelbling · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…