2021

Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping

Kaushik, Prakhar, Gain, Alex, Kortylewski, Adam et al.

Understand

Catastrophic forgetting in neural networks is a significant problem for continual learning.

A majority of the current methods replay previous data during training, which violates the constraints of an ideal continual learning system.
Additionally, current approaches that deal with forgetting ignore the problem of catastrophic remembering, i.e.
the worsening ability to discriminate between data from different tasks.

Built on

Using semi-distributed representations to overcome catastrophic forgetting in connectionist networks
French, R. M · 1991
Earlier work this paper cites.
Catastrophic forgetting in neural networks: the role of rehearsal mechanisms
Robins, A · 1993
Earlier work this paper cites.
Catastrophic forgetting in neural networks: the role of rehearsal mechanisms
Robins, A · 1993
Earlier work this paper cites.
Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference, 1994
French, R · 1994
Earlier work this paper cites.
10 - catastrophic interference in neural networks: Causes, solutions, and data
Lewandowsky, S. and Li, S.-C · 1995
Earlier work this paper cites.
Catastrophic forgetting in connectionist networks
French, R. M · 1999
Earlier work this paper cites.
Laplace propagation
Smola, A., Vishwanathan, V., and Eskin, E · 2003
Earlier work this paper cites.
A framework for learning predictive structures from multiple tasks and unlabeled data
Ando, R. K. and Zhang, T · 2005
Earlier work this paper cites.
Optimal Continual Learning has Perfect Memory and is NP-hard
Original
Knoblauch, J., Husain, H., and Diethe, T · 2006
Earlier work this paper cites.
ImageNet: A Large-Scale Hierarchical Image Database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A · 2012
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A · 2012
Earlier work this paper cites.
The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects
Mermillod, M., Bugaiska, A., and BONIN, P · 2013
Earlier work this paper cites.
Gaussian process multi-task learning using joint feature selection
Srijith, P. K. and Shevade, S · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization, 2014
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Deep residual learning for image recognition, 2015
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Earlier work this paper cites.
Siamese neural networks for one-shot image recognition
Koch, G., Zemel, R., and Salakhutdinov, R · 2015
Earlier work this paper cites.
Human-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B · 2015
Earlier work this paper cites.
Deep residual learning for image recognition, 2015
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Earlier work this paper cites.
A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2016
Hendrycks, D. and Gimpel, K · 2016
Earlier work this paper cites.

Similar

Fearnet: Brain-inspired model for incremental learning
Kemker, R. and Kanan, C · 2017
Cited alongside, same era.
Measuring catastrophic forgetting in neural networks, 2017
Kemker, R., McClure, M., Abitino, A., Hayes, T., and Kanan, C · 2017
Cited alongside, same era.
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al · 2017
Cited alongside, same era.
Learning without forgetting
Li, Z. and Hoiem, D · 2017
Cited alongside, same era.
Variational continual learning, 2017
Nguyen, C. V., Li, Y., Bui, T. D., and Turner, R. E · 2017
Cited alongside, same era.
Continual learning through synaptic intelligence, 2017
Zenke, F., Poole, B., and Ganguli, S · 2017
Cited alongside, same era.
To prune, or not to prune: exploring the efficacy of pruning for model compression
Original
Zhu, M. and Gupta, S · 2017
Cited alongside, same era.
Fearnet: Brain-inspired model for incremental learning
Kemker, R. and Kanan, C · 2017
Cited alongside, same era.
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al · 2017
Cited alongside, same era.
Variational continual learning, 2017
Nguyen, C. V., Li, Y., Bui, T. D., and Turner, R. E · 2017
Cited alongside, same era.
Memory Aware Synapses: Learning What (not) to Forget
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T · 2018
Cited alongside, same era.
Riemannian walk for incremental learning: Understanding forgetting and intransigence
Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, P. H. S · 2018
Cited alongside, same era.

Then

Overcoming catastrophic forgetting with hard attention to the task
Original
Serr, Suris, Miron, and Karatzoglou · 2018
Later among the works it cites.
The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019
Frankle, J. and Carbin, M · 2019
Later among the works it cites.
Continual learning with hypernetworks
Oswald, J. v., Henning, C., Sacramento, J., and Grewe, B. F · 2019
Later among the works it cites.
The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019
Frankle, J. and Carbin, M · 2019
Later among the works it cites.
Uncertainty-guided Continual Learning with Bayesian Neural Networks
Ebrahimi, S., Elhoseiny, M., Darrell, T., and Rohrbach, M · 2020
Later among the works it cites.
Improved schemes for episodic memory-based lifelong learning, 2020
Guo, Y., Liu, M., Yang, T., and Rosing, T · 2020
Later among the works it cites.
Continual learning with node-importance based adaptive group sparse regularization, 2020
Jung, S., Ahn, H., Cha, S., and Moon, T · 2020
Later among the works it cites.
Continual Learning with Bayesian Neural Networks for Non-Stationary Data
Kurle, R., Cseke, B., Klushyn, A., Smagt, P. v. d., and Günnemann, S · 2020
Later among the works it cites.
Continual Learning With Extended Kronecker-Factored Approximate Curvature
Lee, J., Hong, H. G., Joo, D., and Kim, J · 2020
Later among the works it cites.
A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning, 2020
Mundt, M., Hong, Y. W., Pliushch, I., and Ramesh, V · 2020
Later among the works it cites.
Functional Regularisation for Continual Learning with Gaussian Processes
Titsias, M. K., Schwarz, J., Matthews, A. G. d. G., Pascanu, R., and Teh, Y. W · 2020
Later among the works it cites.
SNOW: Subscribing to Knowledge via Channel Pooling for Transfer & Lifelong Learning of Convolutional Neural Networks
Yoo, C., Kang, B., and Cho, M · 2020
Later among the works it cites.
Improved schemes for episodic memory-based lifelong learning, 2020
Guo, Y., Liu, M., Yang, T., and Rosing, T · 2020
Later among the works it cites.
Continual learning with node-importance based adaptive group sparse regularization, 2020
Jung, S., Ahn, H., Cha, S., and Moon, T · 2020
Later among the works it cites.
Continual Learning With Extended Kronecker-Factored Approximate Curvature
Lee, J., Hong, H. G., Joo, D., and Kim, J · 2020
Later among the works it cites.
Functional Regularisation for Continual Learning with Gaussian Processes
Titsias, M. K., Schwarz, J., Matthews, A. G. d. G., Pascanu, R., and Teh, Y. W · 2020
Later among the works it cites.
SNOW: Subscribing to Knowledge via Channel Pooling for Transfer & Lifelong Learning of Convolutional Neural Networks
Yoo, C., Kang, B., and Cho, M · 2020
Later among the works it cites.
Continual deep learning by functional regularisation of memorable past, 2021
Pan, P., Swaroop, S., Immer, A., Eschenhagen, R., Turner, R. E., and Khan, M. E · 2021
Closest in time.
Continual deep learning by functional regularisation of memorable past, 2021
Pan, P., Swaroop, S., Immer, A., Eschenhagen, R., Turner, R. E., and Khan, M. E · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…