2019

Ray Interference: a Source of Plateaus in Deep Reinforcement Learning

Schaul, Tom, Borsa, Diana, Modayil, Joseph et al.

Understand

Rather than proposing a new method, this paper investigates an issue present in existing learning algorithms.

We study the learning dynamics of reinforcement learning (RL), specifically a characteristic coupling between learning and data generation that arises because RL agents control their future data distribution.
In the presence of function approximation, this coupling can lead to a problematic type of 'ray interference', characterized by learning dynamics that sequentially traverse a number of performance plateaus, effectively constraining the agent to learn one thing at a time even when learning in parallel is better.
We establish the conditions under which ray interference occurs, show its relation to saddle points and obtain the exact learning dynamics in a restricted setting.

Built on

Simple statistical gradient-following algorithms for connectionist reinforcement learning
R. J. Williams · 1992
Earlier work this paper cites.
Continual learning in reinforcement environments
M. B. Ring · 1994
Earlier work this paper cites.
Natural gradient works efficiently in learning
S.-I. Amari · 1998
Earlier work this paper cites.
Efficient backprop
Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller · 1998
Earlier work this paper cites.
Fitness sharing and niching methods revisited
B. Sareni and L. Krahenbuhl · 1998
Earlier work this paper cites.
Multi-value-functions: Effcient automatic action hierarchies for multiple goal MDPs
A. W. Moore, L. Baird, and L. P. Kaelbling · 1999
Earlier work this paper cites.
Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning
R. S. Sutton, D. Precup, and S. Singh · 1999
Earlier work this paper cites.
Toward hierarchical decomposition for planning in uncertain environments
T. Lane and L. P. Kaelbling · 2001
Earlier work this paper cites.
Natural actor-critic
J. Peters, S. Vijayakumar, and S. Schaal · 2005
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
X. Glorot and Y. Bengio · 2010
Earlier work this paper cites.
Lifelong machine learning systems: Beyond learning algorithms
D. L. Silver, Q. Yang, and L. Li · 2013
Earlier work this paper cites.
Empirical evaluation of gated recurrent neural networks on sequence modeling
Original
J. Chung, C. Gulcehre, K. Cho, and Y. Bengio · 2014
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
A. M. Saxe, J. L. McClelland, and S. Ganguli · 2014
Earlier work this paper cites.
Deep networks with internal selective attention through feedback connections
M. F. Stollenga, J. Masci, F. Gomez, and J. Schmidhuber · 2014
Earlier work this paper cites.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Original
K. He, X. Zhang, S. Ren, and J. Sun · 2015
Earlier work this paper cites.
Unifying distillation and privileged information
Original
D. Lopez-Paz, L. Bottou, B. Schölkopf, and V. Vapnik · 2015
Earlier work this paper cites.

Similar

Human-level control through deep reinforcement learning
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis · 2015
Cited alongside, same era.
Policy distillation
Original
A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell · 2015
Cited alongside, same era.
Prioritized experience replay
Original
T. Schaul, J. Quan, I. Antonoglou, and D. Silver · 2015
Cited alongside, same era.
Modular active curiosity-driven discovery of tool use
S. Forestier and P.-Y. Oudeyer · 2016
Cited alongside, same era.
Reinforcement learning with unsupervised auxiliary tasks
Original
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu · 2016
Cited alongside, same era.
Mastering the game of Go with deep neural networks and tree search
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis · 2016
Cited alongside, same era.
Tensor switching networks
C.-Y. Tsai, A. M. Saxe, and D. Cox · 2016
Cited alongside, same era.
Pathnet: Evolution channels gradient descent in super neural networks
Original
C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. A. Rusu, A. Pritzel, and D. Wierstra · 2017
Cited alongside, same era.
Genetic policy optimization
Original
T. Gangwani and J. Peng · 2017
Cited alongside, same era.
Automated curriculum learning for neural networks
Original
A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu · 2017
Cited alongside, same era.
Population based training of neural networks
Original
M. Jaderberg, V. Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, et al · 2017
Cited alongside, same era.
Overcoming catastrophic forgetting in neural networks
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al · 2017
Cited alongside, same era.

Then

Gradient diversity empowers distributed learning
Original
D. Yin, A. Pananjady, M. Lam, D. S. Papailiopoulos, K. Ramchandran, and P. Bartlett · 2017
Later among the works it cites.
RUDDER: Return decomposition for delayed rewards
Original
J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, J. Brandstetter, and S. Hochreiter · 2018
Later among the works it cites.
Adapting auxiliary losses using gradient similarity
Original
Y. Du, W. M. Czarnecki, S. M. Jayakumar, R. Pascanu, and B. Lakshminarayanan · 2018
Later among the works it cites.
IMPALA: Scalable distributed Deep-RL with importance weighted actor-learner architectures
Original
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al · 2018
Later among the works it cites.
Dynamic task prioritization for multitask learning
M. Guo, A. Haque, D.-A. Huang, S. Yeung, and L. Fei-Fei · 2018
Later among the works it cites.
Rainbow: Combining improvements in deep reinforcement learning
M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver · 2018
Later among the works it cites.
Multi-task deep reinforcement learning with popart
Original
M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. van Hasselt · 2018
Later among the works it cites.
Openai five
OpenAI · 2018
Later among the works it cites.
Learning dexterous in-hand manipulation
Original
OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. W. Pachocki, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba · 2018
Later among the works it cites.
The Barbados 2018 list of open issues in continual learning
Original
T. Schaul, H. van Hasselt, J. Modayil, M. White, A. White, P. Bacon, J. Harb, S. Mourad, M. G. Bellemare, and D. Precup · 2018
Later among the works it cites.
A general reinforcement learning algorithm that masters chess, shogi, and go through self-play
D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al · 2018
Later among the works it cites.
Reinforcement learning: An introduction
R. S. Sutton and A. G. Barto · 2018
Later among the works it cites.
The value function polytope in reinforcement learning
Original
R. Dadashi, A. A. Taïga, N. L. Roux, D. Schuurmans, and M. G. Bellemare · 2019
Closest in time.
Learning to learn without forgetting by maximizing transfer and minimizing interference
M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, , and G. Tesauro · 2019
Closest in time.
AlphaStar: Mastering the Real-Time Strategy Game StarCraft II
O. Vinyals, I. Babuschkin, J. Chung, M. Mathieu, M. Jaderberg, W. M. Czarnecki, A. Dudzik, A. Huang, P. Georgiev, R. Powell, T. Ewalds, D. Horgan, M. Kroiss, I. Danihelka, J. Agapiou, J. Oh, V. Dalibard, D. Choi, L. Sifre, Y. Sulsky, S. Vezhnevets, J. Molloy, T. Cai, D. Budden, T. Paine, C. Gulcehre, Z. Wang, T. Pfaff, T. Pohlen, Y. Wu, D. Yogatama, J. Cohen, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, C. Apps, K. Kavukcuoglu, D. Hassabis, and D. Silver · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…