2018

Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence

Chaudhry, Arslan, Dokania, Puneet K., Ajanthan, Thalaiyasingam et al.

Understand

Incremental learning (IL) has received a lot of attention recently, however, the literature lacks a precise problem definition, proper evaluation settings, and metrics tailored specifically for the IL problem.

One of the main objectives of this work is to fill these gaps so as to provide a common ground for better understanding of IL.
The main challenge for an IL algorithm is to update the classifier whilst preserving existing knowledge.
We observe that, in addition to forgetting, a known issue while preserving knowledge, IL also suffers from a problem we call intransigence, inability of a model to update its knowledge.

Built on

Kullback, S., Leibler, R.A.: On information and sufficiency. The Annals of Mathematical Statistics (1951)
1951
Earlier work this paper cites.
Hecht-Nielsen, R., et al.: Theory of the backpropagation neural network. Neural Networks 1
1988
Earlier work this paper cites.
Amari, S.I.: Natural gradient works efficiently in learning. Neural Computation (1998)
1998
Earlier work this paper cites.
LeCun, Y.: The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ (1998)
1998
Earlier work this paper cites.
Lee, J.M.: Riemannian manifolds: an introduction to curvature, vol. 176. Springer Science & Business Media (2006)
2006
Earlier work this paper cites.
Le Roux, N., Pierre-Antoine, M., Bengio, Y.: Topmoumoute online natural gradient algorithm. In: NIPS (2007)
2007
Earlier work this paper cites.
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. https://www.cs.toronto.edu/ kriz/cifar.html (2009)
2009
Earlier work this paper cites.
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS (2014)
2014
Earlier work this paper cites.
Pascanu, R., Bengio, Y.: Revisiting natural gradient for deep networks. In: ICLR (2014)
2014
Earlier work this paper cites.

Similar

Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)
2015
Cited alongside, same era.
Martens, J., Grosse, R.: Optimizing neural networks with kronecker-factored approximate curvature. In: ICML (2015)
2015
Cited alongside, same era.
Terekhov, A.V., Montone, G., O’Regan, J.K.: Knowledge transfer in deep block-modular neural networks. In: Conference on Biomimetic and Biohybrid Systems. pp. 268–279 (2015)
2015
Cited alongside, same era.
Grosse, R., Martens, J.: A kronecker-factored approximate fisher matrix for convolution layers. In: ICML (2016)
2016
Cited alongside, same era.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
2016
Cited alongside, same era.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N.C., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America (PNAS) (2016)
2016
Cited alongside, same era.
Li, Z., Hoiem, D.: Learning without forgetting. In: ECCV (2016)
2016
Cited alongside, same era.
Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016)
Original
2016
Cited alongside, same era.
Lee, S.W., Kim, J.H., Ha, J.W., Zhang, B.T.: Overcoming catastrophic forgetting by incremental moment matching. In: NIPS (2017)
2017
Cited alongside, same era.

Then

Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continuum learning. In: NIPS (2017)
2017
Later among the works it cites.
Rebuffi, S.A., Bilen, H., Vedaldi, A.: Learning multiple visual domains with residual adapters. In: NIPS (2017)
2017
Later among the works it cites.
Rebuffi, S.V., Kolesnikov, A., Lampert, C.H.: iCaRL: Incremental classifier and representation learning. In: CVPR (2017)
2017
Later among the works it cites.
Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: NIPS (2017)
2017
Later among the works it cites.
Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: ICML (2017)
Original
2017
Later among the works it cites.
Nguyen, C.V., Li, Y., Bui, T.D., Turner, R.E.: Variational continual learning. ICLR (2018)
2018
Closest in time.
Schwarz, J., Luketina, J., Czarnecki, W.M., Grabska-Barwinska, A., Teh, Y.W., Pascanu, R., Hadsell, R.: Progress & compress: A scalable framework for continual learning. In: ICML (2018)
2018
Closest in time.
Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. In: ICLR (2018)
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…