Conditional Computation for Continual Learning

Using semi-distributed representations to overcome catastrophic forgetting in connectionist networks

Robert M French · 1991

Earlier work this paper cites.

Catastrophic forgetting, rehearsal and pseudorehearsal

Anthony Robins · 1995

Earlier work this paper cites.

Estimating or propagating gradients through stochastic neurons for conditional computation

Original

Yoshua Bengio, Nicholas Léonard, and Aaron Courville · 2013

Earlier work this paper cites.

An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks

I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio · 2013

Earlier work this paper cites.

Maxout networks

Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio · 2013

Cited alongside, same era.

Compete to compute

Original

Rupesh K Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, and Jürgen Schmidhuber · 2013

Cited alongside, same era.

Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning

Original

Kyunghyun Cho and Yoshua Bengio · 2014

Cited alongside, same era.

On the number of linear regions of deep neural networks

Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio · 2014

Cited alongside, same era.

Conditional computation in neural networks for faster models

Original

Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup · 2015

Later among the works it cites.

Overcoming catastrophic forgetting in neural networks

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al · 2017

Later among the works it cites.

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

Original

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean · 2017

Later among the works it cites.