2020

Class-incremental learning: survey and performance evaluation on image classification

Masana, Marc, Liu, Xialei, Twardowski, Bartlomiej et al.

Understand

For future learning systems, incremental learning is desirable because it allows for: efficient resource usage by eliminating the need to retrain from scratch at the arrival of new data; reduced memory usage by preventing or limiting the amount of data required to be stored -- also important when privacy limitations are imposed; and learning that more closely resembles human learning.

The main challenge for incremental learning is catastrophic forgetting, which refers to the precipitous drop in performance on previously learned tasks after learning a new one.
Incremental learning of deep neural networks has seen explosive growth in recent years.
Initial work focused on task-incremental learning, where a task-ID is provided at inference time.

Built on

J. Schmidhuber, “Evolutionary principles in self-referential learning,” Diploma thesis, Tech. Univ. Munich , 1987
1987
Earlier work this paper cites.
M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in Psychology of learning and motivation , 1989
1989
Earlier work this paper cites.
R. Ratcliff, “Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.” Psychological review , 1990
1990
Earlier work this paper cites.
A. W. Moore and C. G. Atkeson, “Prioritized sweeping: Reinforcement learning with less data and less time,” Machine learning , 1993
1993
Earlier work this paper cites.
S. Thrun, “A lifelong learning perspective for mobile robot control,” in Intelligent Robots and Systems , 1995
1995
Earlier work this paper cites.
S. Thrun, “Is learning the n-th thing any easier than learning the first?” in Proc. Adv. Neural Information Processing Systems , 1996
1996
Earlier work this paper cites.
R. M. French, “Catastrophic forgetting in connectionist networks,” Trends in cognitive sciences , 1999
1999
Earlier work this paper cites.
D. L. Silver and R. E. Mercer, “The task rehearsal method of life-long learning: Overcoming impoverished data,” in Conference of the Canadian Society for Computational Studies of Intelligence , 2002
2002
Earlier work this paper cites.
C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, “Model compres-sion,” in International Conference on Knowledge Discovery and Data Mining , 2006
2006
Earlier work this paper cites.
M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in Indian Conference on Computer Vision, Graphics & Image Processing , 2008
2008
Earlier work this paper cites.
A. Torralba, R. Fergus, and W. T. Freeman, “80 million tiny images: A large data set for nonparametric object and scene recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2008
2008
Earlier work this paper cites.
M. Welling, “Herding dynamical weights to learn,” in Proc. International Conference on Machine Learning , 2009
2009
Earlier work this paper cites.
A. Krizhevsky, “Learning multiple layers of features from tiny images,” Citeseer, Tech. Rep., 2009
2009
Earlier work this paper cites.
A. Quattoni and A. Torralba, “Recognizing indoor scenes,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition , 2009
2009
Earlier work this paper cites.
Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proc. International Conference on Machine Learning , 2009
2009
Earlier work this paper cites.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The caltech-ucsd birds-200-2011 dataset,” California Institute of Technology, Tech. Rep. CNS-TR-2011-001, 2011
2011
Earlier work this paper cites.
B. Yao, X. Jiang, A. Khosla, A. L. Lin, L. Guibas, and L. Fei-Fei, “Human action recognition by learning bases of action attributes and parts,” in Proc. IEEE International Conference on Computer Vision , 2011
2011
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Proc. Adv. Neural Information Processing Systems , 2012
2012
Earlier work this paper cites.
M. Mermillod, A. Bugaiska, and P. Bonin, “The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects,” Frontiers in psychology , 2013
2013
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” in Proc. Adv. Neural Information Processing Systems Workshop on Deep Learning , 2013
2013
Earlier work this paper cites.
R. Coop and I. Arel, “Mitigation of catastrophic forgetting in recurrent neural networks using a fixed expansion layer,” in International Joint Conference on Neural Networks , 2013
2013
Earlier work this paper cites.
J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine-grained categorization,” in Proc. IEEE International Conference on Computer Vision Workshops , 2013
2013
Earlier work this paper cites.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi, “Fine-grained visual classification of aircraft,” eprint arXiv:1306.5151, Tech. Rep., 2013
Original
2013
Earlier work this paper cites.
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” in Proc. International Conference on Learning Representations , 2014
2014
Earlier work this paper cites.
G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in Proc. Adv. Neural Information Processing Systems Workshop on Deep Learning , 2014
2014
Earlier work this paper cites.
T. Xiao, J. Zhang, K. Yang, Y. Peng, and Z. Zhang, “Error-driven incremental learning in deep convolutional neural network for large-scale image classification,” in ACM International Conference on Multimedia , 2014
2014
Earlier work this paper cites.
R. Girshick, “Fast r-cnn,” in IEEE International Conference on Computer Vision , 2015
2015
Earlier work this paper cites.
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Proc. Adv. Neural Information Processing Systems , 2015
2015
Earlier work this paper cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , “Imagenet large scale visual recognition challenge,” International Journal of Computer Vision , 2015
2015
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proc. International Conference on Learning Representations , 2015
2015
Earlier work this paper cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in IEEE Conference on Computer Vision and Pattern Recognition , 2015
2015
Earlier work this paper cites.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” arXiv , 2016
2016
Earlier work this paper cites.
H. Jung, J. Ju, M. Jung, and J. Kim, “Less-forgetting learning in deep neural networks,” arXiv , 2016
2016
Earlier work this paper cites.
T. Chen, I. Goodfellow, and J. Shlens, “Net2net: Accelerating learning via knowledge transfer,” in Proc. International Conference on Learning Representations , 2016
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition , 2016
2016
Earlier work this paper cites.
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition , 2017
2017
Earlier work this paper cites.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,” National Academy of Sciences , 2017
2017
Earlier work this paper cites.
R. Aljundi, P. Chakravarty, and T. Tuytelaars, “Expert gate: Lifelong learning with a network of experts,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition , 2017
2017
Earlier work this paper cites.
D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” in Proc. Adv. Neural Information Processing Systems , 2017
2017
Earlier work this paper cites.
Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017
2017
Earlier work this paper cites.
H. Shin, J. K. Lee, J. Kim, and J. Kim, “Continual learning with deep generative replay,” in Proc. Adv. Neural Information Processing Systems , 2017
2017
Earlier work this paper cites.
F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in Proc. International Conference on Machine Learning , 2017
2017
Earlier work this paper cites.
S.-W. Lee, J.-H. Kim, J. Jun, J.-W. Ha, and B.-T. Zhang, “Overcoming catastrophic forgetting by incremental moment matching,” in Proc. Adv. Neural Information Processing Systems , 2017
2017
Earlier work this paper cites.
A. Rannen, R. Aljundi, M. B. Blaschko, and T. Tuytelaars, “Encoder based lifelong learning,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition , 2017
2017
Earlier work this paper cites.

Similar

S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,” in Proc. International Conference on Learning Representations , 2017
2017
Cited alongside, same era.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proc. IEEE International Conference on Computer Vision , 2017
2017
Cited alongside, same era.
C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. A. Rusu, A. Pritzel, and D. Wierstra, “Pathnet: Evolution channels gradient descent in super neural networks,” arXiv , 2017
2017
Cited alongside, same era.
K. Shmelkov, C. Schmid, and K. Alahari, “Incremental learning of object detectors without catastrophic forgetting,” in Proc. IEEE International Conference on Computer Vision , 2017
2017
Cited alongside, same era.
A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. Torr, “Riemannian walk for incremental learning: understanding forgetting and intransigence,” in Proc. European Conference on Computer Vision , 2018
2018
Cited alongside, same era.
G. M. van de Ven and A. S. Tolias, “Three scenarios for continual learning,” in Proc. Adv. Neural Information Processing Systems Workshop on Continual Learning , 2018
2018
Cited alongside, same era.
Z. Chen and B. Liu, “Lifelong machine learning,” Synthesis Lectures on Artificial Intelligence and Machine Learning , 2018
2018
Cited alongside, same era.
P. McClure, C. Y. Zheng, J. R. Kaczmarzyk, J. A. Lee, S. S. Ghosh, D. Nielson, P. Bandettini, and F. Pereira, “Distributed weight consolidation: a brain segmentation case study,” in Proc. Adv. Neural Information Processing Systems , 2018
2018
Cited alongside, same era.
F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari, “End-to-end incremental learning,” in Proc. European Conference on Computer Vision , 2018
2018
Cited alongside, same era.
J. Schwarz, J. Luketina, W. M. Czarnecki, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell, “Progress & compress: A scalable framework for continual learning,” in Proc. International Conference on Machine Learning , 2018
2018
Cited alongside, same era.
A. Mallya and S. Lazebnik, “Packnet: Adding multiple tasks to a single network by iterative pruning,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition , 2018
2018
Cited alongside, same era.
C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner, “Variational continual learning,” in Proc. International Conference on Learning Representations , 2018
2018
Cited alongside, same era.

Then

R. Aljundi, E. Belilovsky, T. Tuytelaars, L. Charlin, M. Caccia, M. Lin, and L. Page-Caccia, “Online continual learning with maximal interfered retrieval,” in Proc. Adv. Neural Information Processing Systems , 2019
2019
Later among the works it cites.
M. Riemer, T. Klinger, D. Bouneffouf, and M. Franceschini, “Scalable recollections for continual lifelong learning,” in Proc. AAAI Conference Artificial Intelligence , 2019
2019
Later among the works it cites.
M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro, “Learning to learn without forgetting by maximizing transfer and minimizing interference,” in Proc. International Conference on Learning Representations , 2019
2019
Later among the works it cites.
S. Farquhar and Y. Gal, “A unifying bayesian view of continual learning,” in Proc. Adv. Neural Information Processing Systems Workshop on Deep Learning , 2019
2019
Later among the works it cites.
H. Ahn, S. Cha, D. Lee, and T. Moon, “Uncertainty-based continual learning with adaptive regularization,” in Proc. Adv. Neural Information Processing Systems , 2019
2019
Later among the works it cites.
M. Zhai, L. Chen, F. Tung, J. He, M. Nawhal, and G. Mori, “Lifelong gan: Continual learning for conditional image generation,” in Proc. IEEE International Conference on Computer Vision , 2019
2019
Later among the works it cites.
Y. Hao, Y. Fu, Y.-G. Jiang, and Q. Tian, “An end-to-end architecture for class-incremental object detection with knowledge distillation,” in International Conference on Multimedia and Expo , 2019
2019
Later among the works it cites.
U. Michieli and P. Zanuttigh, “Incremental learning techniques for semantic segmentation,” in Proc. IEEE International Conference on Computer Vision Workshops , 2019
2019
Later among the works it cites.
O. Tasar, Y. Tarabalka, and P. Alliez, “Incremental learning for semantic segmentation of large-scale remote sensing data,” IEEE Applied Earth Observations and Remote Sensing , 2019
2019
Later among the works it cites.
M. Schak and A. Gepperth, “A study on catastrophic forgetting in deep lstm networks,” in International Conference on Artificial Neural Networks , 2019
2019
Later among the works it cites.
S. Sodhani, S. Chandar, and Y. Bengio, “Toward training recurrent neural networks for lifelong learning,” Neural Computation , 2019
2019
Later among the works it cites.
D. Rao, F. Visin, A. Rusu, R. Pascanu, Y. W. Teh, and R. Hadsell, “Continual unsupervised representation learning,” in Proc. Adv. Neural Information Processing Systems , 2019
2019
Later among the works it cites.
K. Javed and M. White, “Meta-learning representations for continual learning,” in Proc. Adv. Neural Information Processing Systems , 2019
2019
Later among the works it cites.
R. Aljundi, K. Kelchtermans, and T. Tuytelaars, “Task-free continual learning,” in IEEE Conference on Computer Vision and Pattern Recognition , 2019
2019
Later among the works it cites.
F. Kunstner, L. Balles, and P. Hennig, “Limitations of the empirical fisher approximation for natural gradient descent,” in Proc. Adv. Neural Information Processing Systems , 2019
2019
Later among the works it cites.
T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez, “Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges,” Information Fusion , 2020
2020
Closest in time.
O. Sharir, B. Peleg, and Y. Shoham, “The cost of training nlp models: A concise overview,” arXiv , 2020
2020
Closest in time.
T. L. Hayes, K. Kafle, R. Shrestha, M. Acharya, and C. Kanan, “Remind your neural network to prevent catastrophic forgetting,” in Proc. European Conference on Computer Vision , 2020
2020
Closest in time.
J. Zhang, J. Zhang, S. Ghosh, D. Li, S. Tasci, L. Heck, H. Zhang, and C.-C. J. Kuo, “Class-incremental learning via deep model consolidation,” in Proc. IEEE Winter Conference on Applications of Computer Vision , 2020
2020
Closest in time.
J. Lee, H. G. Hong, D. Joo, and J. Kim, “Continual learning with extended kronecker-factored approximate curvature,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition , 2020
2020
Closest in time.
Y. Liu, A.-A. Liu, Y. Su, B. Schiele, and Q. Sun, “Mnemonics training: Multi-class incremental learning without forgetting,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition , 2020
2020
Closest in time.
A. Prabhu, P. H. Torr, and P. K. Dokania, “Gdumb: A simple approach that questions our progress in continual learning,” in Proc. European Conference on Computer Vision . Springer, 2020, pp. 524–540
2020
Closest in time.
M. K. Titsias, J. Schwarz, A. G. d. G. Matthews, R. Pascanu, and Y. W. Teh, “Functional regularisation for continual learning with gaussian processes,” in Proc. International Conference on Learning Representations , 2020
2020
Closest in time.
P. Pan, S. Swaroop, A. Immer, R. Eschenhagen, R. E. Turner, and M. E. Khan, “Continual deep learning by functional regularisation of memorable past,” in Proc. Adv. Neural Information Processing Systems , 2020
2020
Closest in time.
E. Belouadah, A. Popescu, and I. Kanellos, “A comprehensive study of class incremental learning algorithms for visual tasks,” Neural Networks , 2020
2020
Closest in time.
S. Ebrahimi, F. Meier, R. Calandra, T. Darrell, and M. Rohrbach, “Adversarial continual learning,” in Proc. European Conference on Computer Vision , 2020
2020
Closest in time.
T. Adel, H. Zhao, and R. E. Turner, “Continual learning with adaptive weights (claw),” in Proc. International Conference on Learning Representations , 2020
2020
Closest in time.
S. Ebrahimi, M. Elhoseiny, T. Darrell, and M. Rohrbach, “Uncertainty-guided continual learning with bayesian neural networks,” in Proc. International Conference on Learning Representations , 2020
2020
Closest in time.
F. Cermelli, M. Mancini, S. R. Bulo, E. Ricci, and B. Caputo, “Modeling the background for incremental learning in semantic
2020
Closest in time.
R. Del Chiaro, B. Twardowski, A. D. Bagdanov, and J. Van de Weijer, “Ratt: Recurrent attention to transient tasks for continual image captioning,” in International Conference on Machine Learning Workshop LifelongML , 2020
2020
Closest in time.
X. Liu, C. Wu, M. Menta, L. Herranz, B. Raducanu, A. D. Bagdanov, S. Jui, and J. van de Weijer, “Generative feature replay for class-incremental learning,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops , 2020
2020
Closest in time.
A. Iscen, J. Zhang, S. Lazebnik, and C. Schmid, “Memory-efficient incremental learning through feature adaptation,” in Proc. European Conference on Computer Vision , 2020
2020
Closest in time.
L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2020
2020
Closest in time.
L. Yu, B. Twardowski, X. Liu, L. Herranz, K. Wang, Y. Cheng, S. Jui, and J. v. d. Weijer, “Semantic drift compensation for class-incremental learning,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition , 2020
2020
Closest in time.
J. Rajasegaran, S. Khan, M. Hayat, F. S. Khan, and M. Shah, “iTAML: An incremental task-agnostic meta-learning approach,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition , 2020
2020
Closest in time.
M. De Lange and T. Tuytelaars, “Continual prototype evolution: Learning online from non-stationary data streams,” in Proc. IEEE International Conference on Computer Vision , 2020
2020
Closest in time.
M. Masana, B. Twardowski, and J. Van de Weijer, “On class orderings for incremental learning,” in International Conference on Machine Learning Workshop on Continual Learning , 2020
2020
Closest in time.
S. Zagoruyko and N. Komodakis, “Wide residual networks,” in Proc. British Machine Vision Conference , 2016
2020
Closest in time.
M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021
2021
Closest in time.
M. Masana, T. Tuytelaars, and J. van de Weijer, “Ternary feature masks: continual learning without any forgetting,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops , 2021
2021
Closest in time.
A. Chaudhry, A. Gordo, P. K. Dokania, P. Torr, and D. Lopez-Paz, “Using hindsight to anchor past knowledge in continual learning,” in Proc. AAAI Conference Artificial Intelligence , 2021
2021
Closest in time.
S. Li, Y. Du, G. M. van de Ven, A. Torralba, and I. Mordatch, “Energy-based models for continual learning,” in Proc. International Conference on Learning Representations , 2021
2021
Closest in time.
Z. Mai, R. Li, J. Jeong, D. Quispe, H. Kim, and S. Sanner, “Online continual learning in image classification: An empirical survey,” Neurocomputing , vol. 469, pp. 28–51, 2022
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…