2019

Toward Understanding Catastrophic Forgetting in Continual Learning

Nguyen, Cuong V., Achille, Alessandro, Lam, Michael et al.

Understand

We study the relationship between catastrophic forgetting and properties of task sequences.

In particular, given a sequence of tasks, we would like to understand which properties of this sequence influence the error rates of continual learning algorithms trained on the sequence.
To this end, we propose a new procedure that makes use of recent developments in task space modeling as well as correlation analysis to specify and analyze the properties we are interested in.
As an application, we apply our procedure to study two properties of a task sequence: (1) total complexity and (2) sequential heterogeneity.

Built on

A case study of incremental concept induction
J. C. Schlimmer and D. Fisher · 1986
Earlier work this paper cites.
Catastrophic interference in connectionist networks: The sequential learning problem
M. McCloskey and N. J. Cohen · 1989
Earlier work this paper cites.
Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions
R. Ratcliff · 1990
Earlier work this paper cites.
Online learning with random representations
R. S. Sutton and S. D. Whitehead · 1993
Earlier work this paper cites.
Catastrophic forgetting, rehearsal and pseudorehearsal
A. Robins · 1995
Earlier work this paper cites.
Avoiding catastrophic forgetting by coupling two reverberating neural networks
B. Ans and S. Rousset · 1997
Earlier work this paper cites.
CHILD: A first step towards continual learning
M. B. Ring · 1997
Earlier work this paper cites.
Catastrophic forgetting in connectionist networks
R. M. French · 1999
Earlier work this paper cites.
Neural networks with a self-refreshing memory: Knowledge transfer in sequential learning tasks without catastrophic forgetting
B. Ans and S. Rousset · 2000
Earlier work this paper cites.
Laplace propagation
A. J. Smola, S. Vishwanathan, and E. Eskin · 2004
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky and G. Hinton · 2009
Earlier work this paper cites.
MNIST handwritten digit database
Y. LeCun, C. Cortes, and C. Burges · 2010
Earlier work this paper cites.
An automated measure of MDP similarity for transfer in reinforcement learning
H. B. Ammar, E. Eaton, M. E. Taylor, D. C. Mocanu, K. Driessens, G. Weiss, and K. Tuyls · 2014
Earlier work this paper cites.
An empirical investigation of catastrophic forgetting in gradient-based neural networks
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio · 2014
Earlier work this paper cites.
CNN features off-the-shelf: An astounding baseline for recognition
A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
D. P. Kingma and J. Ba · 2015
Earlier work this paper cites.
Lifelong Machine Learning
Z. Chen and B. Liu · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Earlier work this paper cites.

Similar

Progressive neural networks
Original
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell · 2016
Cited alongside, same era.
Towards a neural statistician
H. Edwards and A. Storkey · 2017
Cited alongside, same era.
Deep generative dual memory network for continual learning
Original
N. Kamra, U. Gupta, and Y. Liu · 2017
Cited alongside, same era.
Overcoming catastrophic forgetting in neural networks
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell · 2017
Cited alongside, same era.
Overcoming catastrophic forgetting by incremental moment matching
S.-W. Lee, J.-H. Kim, J. Jun, J.-W. Ha, and B.-T. Zhang · 2017
Cited alongside, same era.
Gradient episodic memory for continual learning
D. Lopez-Paz and M. Ranzato · 2017
Cited alongside, same era.
Learning to select data for transfer learning with Bayesian optimization
S. Ruder and B. Plank · 2017
Cited alongside, same era.
Continual learning with deep generative replay
H. Shin, J. K. Lee, J. Kim, and J. Kim · 2017
Cited alongside, same era.
Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms
Original
H. Xiao, K. Rasul, and R. Vollgraf · 2017
Cited alongside, same era.
Continual learning through synaptic intelligence
F. Zenke, B. Poole, and S. Ganguli · 2017
Cited alongside, same era.
The iMaterialist challenge 2017 dataset
X. Zhang, Y. Cui, Y. Song, H. Adam, and S. Belongie · 2017
Cited alongside, same era.
Life-long disentangled representation learning with cross-domain latent homologies
A. Achille, T. Eccles, L. Matthey, C. Burgess, N. Watters, A. Lerchner, and I. Higgins · 2018
Cited alongside, same era.

Then

Bayesian incremental learning for deep neural networks
M. Kochurov, T. Garipov, D. Podoprikhin, D. Molchanov, A. Ashukha, and D. Vetrov · 2018
Later among the works it cites.
Learning without forgetting
Z. Li and D. Hoiem · 2018
Later among the works it cites.
Variational continual learning
C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner · 2018
Later among the works it cites.
Continual lifelong learning with neural networks: A review
Original
G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter · 2018
Later among the works it cites.
Online structured Laplace approximations for overcoming catastrophic forgetting
Original
H. Ritter, A. Botev, and D. Barber · 2018
Later among the works it cites.
Towards a natural benchmark for continual learning
J. Schwarz, D. Altman, A. Dudzik, O. Vinyals, Y. W. Teh, and R. Pascanu · 2018
Later among the works it cites.
Progress & compress: A scalable framework for continual learning
J. Schwarz, J. Luketina, W. M. Czarnecki, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell · 2018
Later among the works it cites.
Overcoming catastrophic forgetting with hard attention to the task
J. Serrà, D. Surís, M. Miron, and A. Karatzoglou · 2018
Later among the works it cites.
Improving and understanding variational continual learning
S. Swaroop, C. V. Nguyen, T. D. Bui, and R. E. Turner · 2018
Later among the works it cites.
Natural variational continual learning
H. Tseran, M. E. Khan, T. Harada, and T. D. Bui · 2018
Later among the works it cites.
Lifelong learning with dynamically expandable networks
J. Yoon, E. Yang, J. Lee, and S. J. Hwang · 2018
Later among the works it cites.
Taskonomy: Disentangling task transfer learning
A. R. Zamir, A. Sax, W. Shen, L. Guibas, J. Malik, and S. Savarese · 2018
Later among the works it cites.
Task agnostic continual learning using online variational Bayes
C. Zeno, I. Golan, E. Hoffer, and D. Soudry · 2018
Later among the works it cites.
Task2Vec: Task embedding for meta-learning
Original
A. Achille, M. Lam, R. Tewari, A. Ravichandran, S. Maji, C. Fowlkes, S. Soatto, and P. Perona · 2019
Closest in time.
notMNIST dataset
Y. Bulatov · 2019
Closest in time.
Efficient lifelong learning with A-GEM
A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny · 2019
Closest in time.
Transferability and hardness of supervised classification tasks
A. T. Tran, C. V. Nguyen, and T. Hassner · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…