2021

Introducing Symmetries to Black Box Meta Reinforcement Learning

Kirsch, Louis, Flennerhag, Sebastian, van Hasselt, Hado et al.

Understand

Meta reinforcement learning (RL) attempts to discover new RL algorithms automatically from environment interaction.

In so-called black-box approaches, the policy and the learning algorithm are jointly represented by a single neural network.
These methods are very flexible, but they tend to underperform in terms of generalisation to new, unseen environments.
In this paper, we explore the role of symmetries in meta-generalisation.

Built on

Neural network model for a mechanism of pattern recognition unaffected by shift in position-Neocognitron
Fukushima, K. 1979 · 1979
Earlier work this paper cites.
On the optimization of a synaptic learning rule
Bengio, S.; Bengio, Y.; Cloutier, J.; and Gecsei, J. 1992 · 1992
Earlier work this paper cites.
Learning to control fast-weight memories: An alternative to dynamic recurrent networks
Schmidhuber, J. 1992 · 1992
Earlier work this paper cites.
Adapting bias by gradient descent: An incremental version of delta-bar-delta
Sutton, R. S. 1992 · 1992
Earlier work this paper cites.
Learning to learn using gradient descent
Hochreiter, S.; Younger, A. S.; and Conwell, P. R. 2001 · 2001
Earlier work this paper cites.
Natural evolution strategies
Wierstra, D.; Schaul, T.; Peters, J.; and Schmidhuber, J. 2008 · 2008
Earlier work this paper cites.
Learning to learn by gradient descent by gradient descent
Andrychowicz, M.; Denil, M.; Gomez, S.; Hoffman, M. W.; Pfau, D.; Schaul, T.; Shillingford, B.; and De Freitas, N. 2016 · 2016
Earlier work this paper cites.
RL 2 : Fast reinforcement learning via slow reinforcement learning
Original
Duan, Y.; Schulman, J.; Chen, X.; Bartlett, P. L.; Sutskever, I.; and Abbeel, P. 2016 · 2016
Earlier work this paper cites.
Learning to reinforcement learn
Original
Wang, J. X.; Kurth-Nelson, Z.; Tirumala, D.; Soyer, H.; Leibo, J. Z.; Munos, R.; Blundell, C.; Kumaran, D.; and Botvinick, M. 2016 · 2016
Earlier work this paper cites.
Geometric deep learning: going beyond euclidean data
Bronstein, M. M.; Bruna, J.; LeCun, Y.; Szlam, A.; and Vandergheynst, P. 2017 · 2017
Earlier work this paper cites.

Similar

Model-agnostic meta-learning for fast adaptation of deep networks
Finn, C.; Abbeel, P.; and Levine, S. 2017 · 2017
Cited alongside, same era.
Evolution strategies as a scalable alternative to reinforcement learning
Original
Salimans, T.; Ho, J.; Chen, X.; Sidor, S.; and Sutskever, I. 2017 · 2017
Cited alongside, same era.
Deep sets
Zaheer, M.; Kottur, S.; Ravanbakhsh, S.; Poczos, B.; Salakhutdinov, R. R.; and Smola, A. J. 2017 · 2017
Cited alongside, same era.
Evolved policy gradients
Houthooft, R.; Chen, Y.; Isola, P.; Stadie, B.; Wolski, F.; Jonathan Ho, O.; and Abbeel, P. 2018 · 2018
Cited alongside, same era.
Differentiable plasticity: training plastic neural networks with backpropagation
Miconi, T.; Stanley, K.; and Clune, J. 2018 · 2018
Cited alongside, same era.
Meta-Gradient Reinforcement Learning
Xu, Z.; van Hasselt, H. P.; and Silver, D. 2018 · 2018
Cited alongside, same era.
Understanding and correcting pathologies in the training of learned optimizers
Metz, L.; Maheswaranathan, N.; Nixon, J.; Freeman, D.; and Sohl-Dickstein, J. 2019 · 2019
Cited alongside, same era.
Meta-learning curiosity algorithms
Alet, F.; Schneider, M. F.; Lozano-Perez, T.; and Kaelbling, L. P. 2020 · 2020
Cited alongside, same era.
Meta learning via learned loss
Bechtle, S.; Molchanov, A.; Chebotar, Y.; Grefenstette, E.; Righetti, L.; Sukhatme, G.; and Meier, F. 2021 · 2020
Cited alongside, same era.
Meta-Learning with Warped Gradient Descent
Flennerhag, S.; Rusu, A. A.; Pascanu, R.; Visin, F.; Yin, H.; and Hadsell, R. 2020 · 2020
Cited alongside, same era.
Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets
Schmidhuber, J. 1993a
Cited in the paper.
A ‘self-referential’weight matrix
Schmidhuber, J. 1993b
Cited in the paper.

Then

Improving Generalization in Meta Reinforcement Learning using Learned Objectives
Kirsch, L.; van Steenkiste, S.; and Schmidhuber, J. 2020 · 2020
Later among the works it cites.
Meta-learning through hebbian plasticity in random networks
Najarro, E.; and Risi, S. 2020 · 2020
Later among the works it cites.
Discovering reinforcement learning algorithms
Oh, J.; Hessel, M.; Czarnecki, W. M.; Xu, Z.; van Hasselt, H. P.; Singh, S.; and Silver, D. 2020 · 2020
Later among the works it cites.
dm_control: Software and tasks for continuous control
Tunyasuvunakool, S.; Muldal, A.; Doron, Y.; Liu, S.; Bohez, S.; Merel, J.; Erez, T.; Lillicrap, T.; Heess, N.; and Tassa, Y. 2020 · 2020
Later among the works it cites.
A comprehensive survey on graph neural networks
Wu, Z.; Pan, S.; Chen, F.; Long, G.; Zhang, C.; and Philip, S. Y. 2020 · 2020
Later among the works it cites.
Meta-Gradient Reinforcement Learning with an Objective Discovered Online
Xu, Z.; van Hasselt, H. P.; Hessel, M.; Oh, J.; Singh, S.; and Silver, D. 2020 · 2020
Later among the works it cites.
Meta learning backpropagation and improving it
Kirsch, L.; and Schmidhuber, J. 2021 · 2021
Closest in time.
Learning Associative Inference Using Fast Weight Memory
Schlag, I.; Munkhdalai, T.; and Schmidhuber, J. 2021 · 2021
Closest in time.
Meta-learning Symmetries by Reparameterization
Zhou, A.; Knowles, T.; and Finn, C. 2021 · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…