2020

A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning

Mundt, Martin, Hong, Yongwon, Pliushch, Iuliia et al.

Understand

Current deep learning methods are regarded as favorable if they empirically perform well on dedicated test sets.

This mentality is seamlessly reflected in the resurfacing area of continual learning, where consecutively arriving data is investigated.
The core challenge is framed as protecting previously acquired representations from being catastrophically forgotten.
However, comparison of individual methods is nevertheless performed in isolation from the real world by monitoring accumulated benchmark test set performance.

Built on

A Mathematical Theory of Communication
Shannon, C.E., 1948 · 1948
Earlier work this paper cites.
The Organization of Behavior; A Neuropsychological Theory
Hebb, D.O., 1949 · 1949
Earlier work this paper cites.
Dropout : A Simple Way to Prevent Neural Networks from Overfitting
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., 2014 · 1958
Earlier work this paper cites.
Adaptation and Learning in Automatic Systems
Tsypkin, Y.Z., 1971 · 1971
Earlier work this paper cites.
Generalization as search
Mitchell, T.M., 1982 · 1982
Earlier work this paper cites.
Estimation of Dependences Based on Empirical Data
Vapnik, V., 1982 · 1982
Earlier work this paper cites.
Clustering to minimize the maximum intercluster distance
Gonzalez, T.F., 1985 · 1985
Earlier work this paper cites.
Dynamic Node Creation in Backpropagation Networks
Ash, T., 1989 · 1989
Earlier work this paper cites.
Catastrophic Interference in Connectionist Networks : The Sequential Learning Problem
McCloskey, M., Cohen, N.J., 1989 · 1989
Earlier work this paper cites.
Learning from hints in neural networks
Abu-Mostafa, Y.S., 1990 · 1990
Earlier work this paper cites.
Training connectionist networks with queries and selective sampling
Atlas, L.E., Cohn, D.A., Ladner, R., El-Sharkawi, M.A., Marks II, R.J., Aggoune, M.E., Park, D.C., 1990 · 1990
Earlier work this paper cites.
Handwritten Character Recognition Using Neural Network Architectures
Matan, O., Kiang, R., Stenard, C.E., Boser, B.E., Denker, J., Henderson, D., Hubbard, W., Jackel, L., LeCun, Y., 1990 · 1990
Earlier work this paper cites.
Connectionist Models of Recognition Memory: Constraints Imposed by Learning and Forgetting Functions
Ratcliff, R., 1990 · 1990
Earlier work this paper cites.
The Strength of Weak Learnability
Schapire, R.E., 1990 · 1990
Earlier work this paper cites.
Rule-injection hints as a means of improving network performance and learning time
Suddarth, S.C., Kergosien, Y.L., 1990 · 1990
Earlier work this paper cites.
Direct Transfer of Learned Information Among Neural Networks
Pratt, L.Y., Mostow, J., Kamm, C.A., 1991 · 1991
Earlier work this paper cites.
Semi-distributed Representations and Catastrophic Forgetting in Connectionist Networks
French, R.M., 1992 · 1992
Earlier work this paper cites.
Information-Based Objective Functions for Active Data Selection
MacKay, D.J.C., 1992 · 1992
Earlier work this paper cites.
Query by committee
Seung, H.S., Opper, M., Sompolinsky, H., 1992 · 1992
Earlier work this paper cites.
On-line learning processes in artificial neural networks
Heskes, T.M., Kappen, B., 1993 · 1993
Earlier work this paper cites.
Discriminability-Based Transfer between Neural Networks
Pratt, L.Y., 1993 · 1993
Earlier work this paper cites.
Novelty detection and neural network validation
Bishop, C.M., 1994 · 1994
Earlier work this paper cites.
A sequential algorithm for training text classifiers
Lewis, D.D., Gale, W.A., 1994 · 1994
Earlier work this paper cites.
Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory
McClelland, J.L., McNaughton, B.L., O’Reilly, R.C., 1995 · 1995
Earlier work this paper cites.
Catastrophic Forgetting, Rehearsal and Pseudorehearsal
Robins, A., 1995 · 1995
Earlier work this paper cites.
Active learning with statistical models
Cohn, D.A., Ghahramani, Z., Jordan, M.I., 1996 · 1996
Earlier work this paper cites.
Avoiding catastrophic forgetting by coupling two reverberating neural networks
Ans, B., Rousset, S., 1997 · 1997
Earlier work this paper cites.
Multitask Learning
Caruana, R., 1997 · 1997
Earlier work this paper cites.
Pseudo-recurrent Connectionist Networks: An Approach to the ’Sensitivity-Stability’ Dilemma
French, R.M., 1997 · 1997
Earlier work this paper cites.
A decision theoretic generalisation of online learning and an application to boosting
Freund, Y., Schapire, R.E., 1997 · 1997
Earlier work this paper cites.
Long Short-Term Memory
Hochreiter, S., Schmidhuber, J., 1997 · 1997
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998 · 1998
Earlier work this paper cites.
Employing EM and Pool-Based Active Learning for Text Classification
McCallum, A.K., Nigam, K., 1998 · 1998
Earlier work this paper cites.
Online Learning and Stochastic Approximations, in: Online Learning in Neural Networks, pp. 9–42
Bottou, L., 1999 · 1999
Earlier work this paper cites.
On-line learning in neural networks
Saad, D. (Ed.), 1999 · 1999
Earlier work this paper cites.
Neurogenesis in the adult brain: Death of a dogma
Gross, C.G., 2000 · 2000
Earlier work this paper cites.
A Probability Analysis on the Value of Unlabelled Data for Classification Problems
Zhang, T., Oles, F.J., 2000 · 2000
Earlier work this paper cites.
Toward optimal active learning through monte carlo estimation of error reduction
Roy, N., McCallum, A., 2001 · 2001
Earlier work this paper cites.
Support Vector Machine Active Learning with Applications to Text Classification
Tong, S., Koller, D., 2001 · 2001
Earlier work this paper cites.
Online Convex Programming and Generalized Infinitesimal Gradient Ascent
Zinkevich, M., 2003 · 2003
Earlier work this paper cites.
Active learning using pre-clustering
Nguyen, H.T., Smeulders, A., 2004 · 2004
Earlier work this paper cites.
Analysis of a greedy active learning strategy
Dasgupta, S., 2005 · 2005
Earlier work this paper cites.
Object classification from a single example utilizing class relevance metrics
Fink, M., 2005 · 2005
Earlier work this paper cites.
Open set face recognition using transduction
Li, F., Wechsler, H., 2005 · 2005
Earlier work this paper cites.
Core vector machines: Fast SVM training on very large data sets
Tsang, I.W., Kwok, J.T., Cheung, P.M., 2005 · 2005
Earlier work this paper cites.
One-shot learning of object categories
Fei-Fei, L., Fergus, R., Perona, P., 2006 · 2006
Earlier work this paper cites.
Inference with the Universum
Weston, J., Collobert, R., Sinz, F., Bottou, L., Vapnik, V., 2006 · 2006
Earlier work this paper cites.
Noise or Signal: The Role of Image Backgrounds in Object Recognition
Xiao, K., Engstrom, L., Ilyas, A., Madry, A., 2020 · 2006
Earlier work this paper cites.
Classification with a reject option using a hinge loss
Bartlett, P.L., Wegkamp, M.H., 2008 · 2008
Earlier work this paper cites.
An analysis of active learning strategies for sequence labeling tasks
Settles, B., Craven, M., 2008 · 2008
Earlier work this paper cites.
Curriculum learning
Bengio, Y., Louradour, J., Collobert, R., Weston, J., 2009 · 2009
Earlier work this paper cites.
Multi-class active learning for image classification
Joshi, A.J., Porikli, F., Papanikolopoulos, N., 2009 · 2009
Earlier work this paper cites.
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A., 2009 · 2009
Earlier work this paper cites.
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C.H., Nickisch, H., Harmeling, S., 2009 · 2009
Earlier work this paper cites.
Herding dynamical weights to learn
Welling, M., 2009 · 2009
Earlier work this paper cites.
Toward an architecture for never-ending language learning
Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka, E.R., Mitchell, T.M., 2010 · 2010
Earlier work this paper cites.
Known knowns, known unknowns and unknown unknowns: A 2010 update on carotid artery disease
Naylor, R., 2010 · 2010
Earlier work this paper cites.
A Survey on Transfer Learning
Pan, S.J., Yang, Q., 2010 · 2010
Earlier work this paper cites.
Online Incremental Feature Learning with Denoising Autoencoders
Zhou, G., Sohn, K., Lee, H., 2012 · 2012
Earlier work this paper cites.

Similar

Neil: Extracting visual knowledge from web data
Chen, X., Shrivastava, A., Gupta, A., 2013 · 2013
Cited alongside, same era.
Auto-Encoding Variational Bayes
Kingma, D.P., Welling, M., 2013 · 2013
Cited alongside, same era.
Adaptive active learning for image classification
Li, X., Guo, Y., 2013 · 2013
Cited alongside, same era.
Towards Open Set Recognition
Scheirer, W.J., Rocha, A., Sapkota, A., Boult, T.E., 2013 · 2013
Cited alongside, same era.
Regulation and function of adult neurogenesis: from genes to cognition
Aimone, J.B., Li, Y., Lee, S.W., Clemenson, G.D., Deng, W., Gage, F.H., 2014 · 2014
Cited alongside, same era.
Generative Adversarial Nets
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2014 · 2014
Cited alongside, same era.
Distilling the Knowledge in a Neural Network
Hinton, G.E., Vinyals, O., Dean, J., 2014 · 2014
Cited alongside, same era.
Learning and transferring mid-level image representations using convolutional neural networks
Oquab, M., Bottou, L., Laptev, I., Sivic, J., 2014 · 2014
Cited alongside, same era.
Probability Models For Open Set Recognition
Scheirer, W.J., Jain, L.P., Boult, T.E., 2014 · 2014
Cited alongside, same era.
Functional neurogenesis in the adult hippocampus: Then and now
Vadodaria, K.C., Jessberger, S., 2014 · 2014
Cited alongside, same era.
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., Lipson, H., 2014 · 2014
Cited alongside, same era.
Coresets for Nonparametric Estimation - the Case of DP-Means
Bachem, O., Lucic, M., Krause, A., 2015 · 2015
Cited alongside, same era.

Then

Revisiting Distillation and Incremental Classifier Learning
Javed, K., Shafait, F., 2018 · 2018
Later among the works it cites.
FearNet: Brain-inspired model for incremental learning
Kemker, R., Kanan, C., 2018 · 2018
Later among the works it cites.
Measuring Catastrophic Forgetting in Neural Networks
Kemker, R., McClure, M., Abitino, A., Hayes, T., Kanan, C., 2018 · 2018
Later among the works it cites.
Enhancing the Reliability of Out-of-distribution Image Detection in Neural Networks
Liang, S., Li, Y., Srikant, R., 2018 · 2018
Later among the works it cites.
Efficient active learning for image classification and segmentation using a sample selection and conditional generative adversarial network
Mahapatra, D., Bozorgtabar, B., Thiran, J.P., Reyes, M., 2018 · 2018
Later among the works it cites.
Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
Mallya, A., Davis, D., Lazebnik, S., 2018 · 2018
Later among the works it cites.
Dropout Sampling for Robust Object Detection in Open-Set Conditions, in: IEEE International Conference on Robotics and Automation (ICRA), pp. 3243–3249
Miller, D., Nicholson, L., Dayoub, F., Sunderhauf, N., 2018 · 2018
Later among the works it cites.
Variational Continual Learning
Nguyen, C.V., Li, Y., Bui, T.D., Turner, R.E., 2018 · 2018
Later among the works it cites.
Experience Replay for Continual Learning
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T.P., Wayne, G., 2018 · 2018
Later among the works it cites.
Online deep learning: Learning deep neural networks on the fly
Sahoo, D., Pham, Q., Lu, J., Hoi, S.C.H., 2018 · 2018
Later among the works it cites.
Active learning for convolutional neural networks: A core-set approach
Sener, O., Savarese, S., 2018 · 2018
Later among the works it cites.
Overcoming Catastrophic forgetting with hard attention to the task
Serra, J., Suris, D., Mirón, M., Karatzoglou, A., 2018 · 2018
Later among the works it cites.
Generative replay with feedback connections as a general strategy for continual learning
Original
van de Ven, G.M., Tolias, A.S., 2018 · 2018
Later among the works it cites.
Memory Replay GANs: learning to generate images from new categories without forgetting
Wu, C., Herranz, L., Liu, X., Wang, Y., van de Weijer, J., Raducanu, B., 2018 · 2018
Later among the works it cites.
Reinforced continual learning
Xu, J., Zhu, Z., 2018 · 2018
Later among the works it cites.
Lifelong Learning with Dynamically Expandable Networks
Yoon, J., Yang, E., Lee, J., Hwang, S.J., 2018 · 2018
Later among the works it cites.
Uncertainty-based Continual Learning with Adaptive Regularization
Ahn, H., Cha, S., Lee, D., Moon, T., 2019 · 2019
Later among the works it cites.
Learning and the Unknown : Surveying Steps Toward Open World Recognition
Boult, T.E., Cruz, S., Dhamija, A.R., Gunther, M., Henrydoss, J., Scheirer, W.J., 2019 · 2019
Later among the works it cites.
Efficient lifelong learning with A-GEM
Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M., 2019 · 2019
Later among the works it cites.
Attract or Distract: Exploit the Margin of Open Set
Feng, Q., Kang, G., Fan, H., Yang, Y., 2019 · 2019
Later among the works it cites.
Deep Active Learning with a Neural Architecture Search
Geifman, Y., El-Yaniv, R., 2019 · 2019
Later among the works it cites.
Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
Geirhos, R., Michaelis, C., Wichmann, F.A., Rubisch, P., Bethge, M., Brendel, W., 2019 · 2019
Later among the works it cites.
Benchmarking neural network robustness to common corruptions and perturbations
Hendrycks, D., Dietterich, T., 2019 · 2019
Later among the works it cites.
Adversarial Examples are not Bugs, they are Features
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A., 2019 · 2019
Later among the works it cites.
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Müller, K.R., 2019 · 2019
Later among the works it cites.
Generative Models from the perspective of Continual Learning
Lesort, T., Caselles-Dupré, H., Garcia-Ortiz, M., Stoian, A., Filliat, D., 2019 · 2019
Later among the works it cites.
Detecting the Unexpected via Image Resynthesis
Lis, K., Nakka, K., Fua, P., Salzmann, M., 2019 · 2019
Later among the works it cites.
Disentangling disentanglement in variational autoencoders
Mathieu, E., Rainforth, T., Siddharth, N., Teh, Y.W., 2019 · 2019
Later among the works it cites.
Open Set Recognition Through Deep Neural Network Uncertainty: Does Out-of-Distribution Detection Require Generative Classifiers?
Mundt, M., Pliushch, I., Majumder, S., Ramesh, V., 2019 · 2019
Later among the works it cites.
Do Deep Generative Models Know What They Don’t Know?
Nalisnick, E., Matsukawa, A., Teh, Y.W., Gorur, D., Lakshminarayanan, B., 2019 · 2019
Later among the works it cites.
Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J.V., Lakshminarayanan, B., Snoek, J., 2019 · 2019
Later among the works it cites.
Continual Lifelong Learning with Neural Networks: A Review
Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S., 2019 · 2019
Later among the works it cites.
Continual Learning by Asymmetric Loss Approximation with Single-Side Overestimation
Park, D., Hong, S., Han, B., Lee, K.M., 2019 · 2019
Later among the works it cites.
OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations
Perera, P., Nallapati, R., Xiang, B., 2019 · 2019
Later among the works it cites.
A Comprehensive, Application-Oriented Study of Catastrophic Forgetting in DNNs
Pfülb, B., Gepperth, A., 2019 · 2019
Later among the works it cites.
Variational adversarial active learning
Sinha, S., Ebrahimi, S., Darrell, T., 2019 · 2019
Later among the works it cites.
Towards Training Recurrent Neural Networks for Lifelong Learning
Sodhani, S., Chandar, S., Bengio, Y., 2019 · 2019
Later among the works it cites.
Bayesian generative active deep learning
Tran, T., Do, T.T., Reid, I., Carneiro, G., 2019 · 2019
Later among the works it cites.
Large Scale Incremental Learning
Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., Fu, Y., 2019 · 2019
Later among the works it cites.
Incremental Learning Using Conditional Adversarial Networks
Xiang, Y., Fu, Y., Ji, P., Huang, H., 2019 · 2019
Later among the works it cites.
Classification-Reconstruction Learning for Open-Set Recognition
Yoshihashi, R., Shao, W., Kawakami, R., You, S., Iida, M., Naemura, T., 2019 · 2019
Later among the works it cites.
Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancy
Yu, Q., Aizawa, K., 2019 · 2019
Later among the works it cites.
Lifelong GAN: Continual Learning for Conditional Image Generation
Zhai, M., Chen, L., Tung, F., He, J., Nawhal, M., Mori, G., 2019 · 2019
Later among the works it cites.
Uncertainty-guided Continual Learning with Bayesian Neural Networks
Ebrahimi, S., Elhoseiny, M., Darrell, T., Rohrbach, M., 2020 · 2020
Closest in time.
Let’s Agree to Agree: Neural Networks Share Classification Order on Real Datasets
Hacohen, G., Choshen, L., Weinshall, D., 2020 · 2020
Closest in time.
Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges
Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., Díaz-Rodríguez, N., 2020 · 2020
Closest in time.
Adversarial sampling for active learning
Mayer, C., Timofte, R., 2020 · 2020
Closest in time.
Human-in-the-Loop Machine Learning
Munro, R., 2020 · 2020
Closest in time.
Deep Active Learning: Unified and Principled Method for Query and Training
Shui, C., Zhou, F., Gagné, C., Wang, B., 2020 · 2020
Closest in time.
Generalizing from a Few Examples: A Survey on Few-Shot Learning
Wang, Y., Yao, Q., Kwok, J., Ni, L.M., 2020 · 2020
Closest in time.
A continual learning survey: Defying forgetting in classification tasks
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., Tuytelaars, T., 2021 · 2021
Closest in time.
A procedural world generation framework for systematic evaluation of continual learning
Hess, T., Mundt, M., Pliushch, I., Ramesh, V., 2021 · 2021
Closest in time.
The clear benchmark: Continual learning on real-world imagery
Lin, Z., Pathak, D., Ramanan, D., 2021 · 2021
Closest in time.
Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition
Mundt, M., Majumder, S., Pliushch, I., Hong, Y.W., Ramesh, V., 2022 · 2022
Closest in time.
When Deep Classifiers Agree: Analyzing Correlations between Learning Order and Image Statistics
Pliushch, I., Mundt, M., Lupp, N., Ramesh, V., 2022 · 2022
Closest in time.
Climb: A continual learning benchmark for vision-and-language tasks
Srinivasan, T., Chang, T.Y., Pinto Alva, L.L., Chochlakis, G., Rostami, M., Thomason, J., 2022 · 2022
Closest in time.
Wild-time: A benchmark of in-the-wild distribution shift over time
Yao, H., Choi, C., Lee, Y., Koh, P., Finn, C., 2022 · 2022
Closest in time.
How well do unsupervised learning algorithms model human real-time and life-long learning?
Zhuang, C., Xiang, V., Bai, Y., Jia, X., Turk-Browne, N., Norman, K., DiCarlo, J., Yamins, D.L., 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…