2019

A continual learning survey: Defying forgetting in classification tasks

De Lange, Matthias, Aljundi, Rahaf, Masana, Marc et al.

Understand

Artificial neural networks thrive in solving the classification problem for a particular rigid task, acquiring knowledge through generalized learning behaviour from a distinct training phase.

The resulting network resembles a static entity of knowledge, with endeavours to extend this knowledge without targeting the original task resulting in a catastrophic forgetting.
Continual learning shifts this paradigm towards networks that can continually accumulate knowledge over different tasks without the need to retrain from scratch.
We focus on task incremental classification, where tasks arrive sequentially and are delineated by clear boundaries.

Built on

S. Grossberg, Studies of mind and brain : neural principles of learning, perception, development, cognition, and motor control , ser. Boston studies in the philosophy of science 70. Dordrecht: Reidel, 1982
1982
Earlier work this paper cites.
M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in Psychology of learning and motivation . Elsevier, 1989, vol. 24, pp. 109–165
1989
Earlier work this paper cites.
R. M. French, “Semi-distributed representations and catastrophic forgetting in connectionist networks,” Connection Science , vol. 4, no. 3-4, pp. 365–377, 1992
1992
Earlier work this paper cites.
J. K. Kruschke, “Alcove: an exemplar-based connectionist model of category learning.” Psychological review , vol. 99, no. 1, p. 22, 1992
1992
Earlier work this paper cites.
S. A. Sloman and D. E. Rumelhart, “Reducing interference in distributed memories through episodic gating,” Essays in honor of WK Estes , vol. 1, pp. 227–248, 1992
1992
Earlier work this paper cites.
D. J. MacKay, “A practical bayesian framework for backpropagation networks,” Neural computation , vol. 4, no. 3, pp. 448–472, 1992
1992
Earlier work this paper cites.
——, “Human category learning: Implications for backpropagation models,” Connection Science , vol. 5, no. 1, pp. 3–36, 1993
1993
Earlier work this paper cites.
J. Rueckl, “Jumpnet: A multiple-memory connectionist architecture,” CogSci , no. 24, pp. 866–871, 1993
1993
Earlier work this paper cites.
——, “Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference,” network , vol. 1111, p. 00001, 1994
1994
Earlier work this paper cites.
A. Robins, “Catastrophic forgetting, rehearsal and pseudorehearsal,” Connection Science , vol. 7, no. 2, pp. 123–146, 1995
1995
Earlier work this paper cites.
R. M. French, “Pseudo-recurrent connectionist networks: An approach to the’sensitivity-stability’dilemma,” Connection Science , vol. 9, no. 4, pp. 353–380, 1997
1997
Earlier work this paper cites.
B. Ans and S. Rousset, “Avoiding catastrophic forgetting by coupling two reverberating neural networks,” Comptes Rendus de l’Académie des Sciences-Series III-Sciences de la Vie , vol. 320, no. 12, pp. 989–997, 1997
1997
Earlier work this paper cites.
L. Bottou, “Online learning and stochastic approximations,” On-line learning in neural networks , vol. 17, no. 9, p. 142, 1998
1998
Earlier work this paper cites.
R. M. French, “Catastrophic forgetting in connectionist networks,” Trends in cognitive sciences , vol. 3, no. 4, pp. 128–135, 1999
1999
Earlier work this paper cites.
D. L. Silver and R. E. Mercer, “The task rehearsal method of life-long learning: Overcoming impoverished data,” in Conference of the Canadian Society for Computational Studies of Intelligence . Springer, 2002, pp. 90–101
2002
Earlier work this paper cites.
M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in ICVGIP . IEEE, 2008, pp. 722–729
2008
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in CVPR . Ieee, 2009, pp. 248–255
2009
Earlier work this paper cites.
A. Quattoni and A. Torralba, “Recognizing indoor scenes,” in CVPR . IEEE, 2009, pp. 413–420
2009
Earlier work this paper cites.
T. E. De Campos, B. R. Babu, M. Varma et al. , “Character recognition in natural images.” VISAPP (2) , vol. 7, 2009
2009
Earlier work this paper cites.
Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in ICML . ACM, 2009, pp. 41–48
2009
Earlier work this paper cites.
S. J. Pan and Q. Yang, “A survey on transfer learning,” TKDE , vol. 22, no. 10, pp. 1345–1359, 2009
2009
Earlier work this paper cites.
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in CVPR , 2017, pp. 2001–2010
2010
Earlier work this paper cites.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona, “Caltech-ucsd birds 200,” 2010
2010
Earlier work this paper cites.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011
2011
Earlier work this paper cites.
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” arXiv preprint arXiv:1207.0580 , 2012
Original
2012
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in NeurIPS , 2012, pp. 1097–1105
2012
Earlier work this paper cites.
S. Shalev-Shwartz et al. , “Online learning and online convex optimization,” Foundations and Trends® in Machine Learning , vol. 4, no. 2, pp. 107–194, 2012
2012
Earlier work this paper cites.
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” arXiv preprint arXiv:1312.6211 , 2013
Original
2013
Earlier work this paper cites.
R. Pascanu and Y. Bengio, “Revisiting natural gradient for deep networks,” arXiv preprint arXiv:1301.3584 , 2013
Original
2013
Earlier work this paper cites.
J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine-grained categorization,” in ICCV workshop , 2013, pp. 554–561
2013
Earlier work this paper cites.
S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine-grained visual classification of aircraft,” arXiv preprint arXiv:1306.5151 , 2013
Original
2013
Earlier work this paper cites.
N. Srivastava, G. Hinton, A. Krizdhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” JMLR , vol. 15, no. 1, pp. 1929–1958, 2014
2014
Earlier work this paper cites.
A. Pentina and C. H. Lampert, “A pac-bayesian bound for lifelong learning.” in ICML , 2014, pp. 991–999
2014
Earlier work this paper cites.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in NeurIPS , 2014, pp. 2672–2680
2014
Earlier work this paper cites.

Similar

Y. Bengio and Y. LeCun, Eds., ICLR, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014. [Online]. Available: https://openreview.net/group?id=ICLR.cc/2014
2014
Cited alongside, same era.
J. Martens, “New insights and perspectives on the natural gradient method,” arXiv preprint arXiv:1412.1193 , 2014
Original
2014
Cited alongside, same era.
I. J. Goodfellow, O. Vinyals, and A. M. Saxe, “Qualitatively characterizing neural network optimization problems,” arXiv preprint arXiv:1412.6544 , 2014
Original
2014
Cited alongside, same era.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , “Imagenet large scale visual recognition challenge,” IJCV , vol. 115, no. 3, pp. 211–252, 2015
2015
Cited alongside, same era.
A. Pentina and C. H. Lampert, “Lifelong learning with non-iid tasks,” in NeurIPS , 2015, pp. 1540–1548
2015
Cited alongside, same era.
M.-F. Balcan, A. Blum, and S. Vempala, “Efficient representations for lifelong learning and autoencoding,” in Conference on Learning Theory , 2015, pp. 191–210
2015
Cited alongside, same era.
G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531 , 2015
Original
2015
Cited alongside, same era.
M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” IJCV , vol. 111, no. 1, pp. 98–136, 2015
2015
Cited alongside, same era.
A. Gepperth and C. Karaoguz, “A bio-inspired incremental learning architecture for applied perceptual problems,” Cognitive Computation , vol. 8, no. 5, pp. 924–934, 2016
2016
Cited alongside, same era.
P. Alquier, T. T. Mai, and M. Pontil, “Regret bounds for lifelong learning,” arXiv preprint arXiv:1610.08628 , 2016
Original
2016
Cited alongside, same era.
Z. Li and D. Hoiem, “Learning without forgetting,” in ECCV . Springer, 2016, pp. 614–629
2016
Cited alongside, same era.
H. Jung, J. Ju, M. Jung, and J. Kim, “Less-forgetting learning in deep neural networks,” arXiv preprint arXiv:1607.00122 , 2016
Original
2016
Cited alongside, same era.

Then

R. Kemker, M. McClure, A. Abitino, T. L. Hayes, and C. Kanan, “Measuring catastrophic forgetting in neural networks,” in AAAI , 2018
2018
Later among the works it cites.
Y.-C. Hsu, Y.-C. Liu, and Z. Kira, “Re-evaluating continual learning scenarios: A categorization and case for strong baselines,” arXiv preprint arXiv:1810.12488 , 2018
Original
2018
Later among the works it cites.
D. Isele and A. Cosgun, “Selective experience replay for lifelong learning,” in AAAI , 2018
2018
Later among the works it cites.
C. Atkinson, B. McCane, L. Szymanski, and A. Robins, “Pseudo-recursal: Solving the catastrophic forgetting problem in deep neural networks,” arXiv preprint arXiv:1802.03875 , 2018
Original
2018
Later among the works it cites.
F. Lavda, J. Ramapuram, M. Gregorova, and A. Kalousis, “Continual classification learning using generative models,” arXiv preprint arXiv:1810.10612 , 2018
Original
2018
Later among the works it cites.
X. Liu, M. Masana, L. Herranz, J. Van de Weijer, A. M. Lopez, and A. D. Bagdanov, “Rotate your networks: Better weight consolidation and less catastrophic forgetting,” in ICPR . IEEE, 2018, pp. 2262–2268
2018
Later among the works it cites.
A. Mallya and S. Lazebnik, “Packnet: Adding multiple tasks to a single network by iterative pruning,” in CVPR , 2018, pp. 7765–7773
2018
Later among the works it cites.
A. Mallya, D. Davis, and S. Lazebnik, “Piggyback: Adapting a single network to multiple tasks by learning to mask weights,” in ECCV , 2018, pp. 67–82
2018
Later among the works it cites.
J. Serrà, D. Surís, M. Miron, and A. Karatzoglou, “Overcoming catastrophic forgetting with hard attention to the task,” arXiv preprint arXiv:1801.01423 , 2018
Original
2018
Later among the works it cites.
J. Xu and Z. Zhu, “Reinforced Continual Learning,” in NeurIPS , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. Curran Associates, Inc., 2018, pp. 899–908. [Online]. Available: http://papers.nips.cc/paper/7369-reinforced-continual-learning.pdf
2018
Later among the works it cites.
C. Zeno, I. Golan, E. Hoffer, and D. Soudry, “Task agnostic continual learning using online variational bayes,” arXiv preprint arXiv:1803.10123 , 2018
Original
2018
Later among the works it cites.
F. Huszár, “Note on the quadratic penalties in elastic weight consolidation,” PNAS , p. 201717042, 2018
2018
Later among the works it cites.
J. Schwarz, J. Luketina, W. M. Czarnecki, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell, “Progress & compress: A scalable framework for continual learning,” arXiv preprint arXiv:1805.06370 , 2018
Original
2018
Later among the works it cites.
“iNaturalist dataset: FGVC5 workshop at CVPR 2018.” [Online]. Available: https://www.kaggle.com/c/inaturalist-2018
2018
Later among the works it cites.
C. Chung, S. Patel, R. Lee, L. Fu, S. Reilly, T. Ho, J. Lionetti, M. D. George, and P. Taylor, “Implementation of an integrated computerized prescriber order-entry system for chemotherapy in a multisite safety-net health system,” AJHP , 2018
2018
Later among the works it cites.
H. Xu, B. Liu, L. Shu, and P. S. Yu, “Learning to accept new classes without training,” arXiv preprint arXiv:1809.06004 , 2018
Original
2018
Later among the works it cites.
A. Bendale and T. Boult, “Towards open world recognition,” in CVPR , 2015, pp. 1893–1902
2018
Later among the works it cites.
G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” Neural Networks , 2019
2019
Closest in time.
T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez, “Continual learning for robotics,” arXiv preprint arXiv:1907.00182 , 2019
Original
2019
Closest in time.
B. Pfülb and A. Gepperth, “A comprehensive, application-oriented study of catastrophic forgetting in dnns,” arXiv preprint arXiv:1905.08101 , 2019
Original
2019
Closest in time.
R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio, “Online continual learning with no task boundaries,” arXiv preprint arXiv:1903.08671 , 2019
Original
2019
Closest in time.
D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experience replay for continual learning,” in Advances in Neural Information Processing Systems , 2019, pp. 350–360
2019
Closest in time.
A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato, “Continual learning with tiny episodic memories,” arXiv preprint arXiv:1902.10486 , 2019
Original
2019
Closest in time.
H. Ahn, S. Cha, D. Lee, and T. Moon, “Uncertainty-based continual learning with adaptive regularization,” in NeurIPS , 2019, pp. 4394–4404
2019
Closest in time.
R. Aljundi, K. Kelchtermans, and T. Tuytelaars, “Task-free continual learning,” in CVPR , 2019, pp. 11 254–11 263
2019
Closest in time.
C. V. Nguyen, A. Achille, M. Lam, T. Hassner, V. Mahadevan, and S. Soatto, “Toward understanding catastrophic forgetting in continual learning,” arXiv preprint arXiv:1908.01091 , 2019
Original
2019
Closest in time.
M. D. Lange, X. Jia, S. Parisot, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “Unsupervised model personalization while preserving privacy and scalability: An open problem,” in CVPR , 2020, pp. 14 463–14 472
2020
Closest in time.
C. S. Lee and A. Y. Lee, “Clinical applications of continual learning machine learning,” The Lancet Digital Health , vol. 2, no. 6, pp. e279–e281, 2020
2020
Closest in time.
P. Gupta, Y. Chaudhary, T. Runkler, and H. Schuetze, “Neural topic modeling with continual lifelong learning,” in ICML . PMLR, 2020, pp. 3907–3917
2020
Closest in time.
M. D. Lange and T. Tuytelaars, “Continual prototype evolution: Learning online from non-stationary data streams,” 2020
2020
Closest in time.
J. Zhang, J. Zhang, S. Ghosh, D. Li, S. Tasci, L. Heck, H. Zhang, and C.-C. J. Kuo, “Class-incremental learning via deep model consolidation,” in WACV , 2020, pp. 1131–1140
2020
Closest in time.
M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. van de Weijer, “Class-incremental learning: survey and performance evaluation,” arXiv preprint arXiv:2010.15277 , 2020
Original
2020
Closest in time.
M. Masana, T. Tuytelaars, and J. van de Weijer, “Ternary feature masks: continual learning without any forgetting,” arXiv preprint arXiv:2001.08714 , 2020
Original
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…