2017

A Strategy for an Uncompromising Incremental Learner

Venkatesan, Ragav, Venkateswara, Hemanth, Panchanathan, Sethuraman et al.

Understand

Multi-class supervised learning systems require the knowledge of the entire range of labels they predict.

Often when learnt incrementally, they suffer from catastrophic forgetting.
To avoid this, generous leeways have to be made to the philosophy of incremental learning that either forces a part of the machine to not learn, or to retrain the machine again with a selection of the historic data.
While these hacks work to various degrees, they do not adhere to the spirit of incremental learning.

Built on

Catastrophic interference in connectionist networks: The sequential learning problem
M. McCloskey and N. J. Cohen · 1989
Earlier work this paper cites.
Catastrophic interference in connectionist networks: Can it be predicted, can it be prevented?
R. M. French · 1993
Earlier work this paper cites.
Types of incremental learning
P. Jantke · 1993
Earlier work this paper cites.
Catastrophic forgetting, rehearsal and pseudorehearsal
A. Robins · 1995
Earlier work this paper cites.
Pseudo-recurrent connectionist networks: An approach to the’sensitivity-stability’dilemma
R. M. French · 1997
Earlier work this paper cites.
Learn++: An incremental learning algorithm for supervised neural networks
R. Polikar, L. Upda, S. S. Upda, and V. Honavar · 2001
Earlier work this paper cites.
An empirical evaluation of deep architectures on problems with many factors of variation
H. Larochelle, D. Erhan, A. Courville, J. Bergstra, and Y. Bengio · 2007
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky and G. Hinton · 2009
Earlier work this paper cites.
Learn++. nc: Combining ensemble of classifiers with dynamically weighted consult-and-vote for efficient incremental learning of new classes
M. D. Muhlbaier, A. Topalis, and R. Polikar · 2009
Earlier work this paper cites.
Herding dynamical weights to learn
M. Welling · 2009
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
V. Nair and G. E. Hinton · 2010
Earlier work this paper cites.

Similar

Reading digits in natural images with unsupervised feature learning
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng · 2011
Cited alongside, same era.
Metric learning for large scale image classification: Generalizing to new classes at near-zero cost
T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka · 2012
Cited alongside, same era.
An empirical investigation of catastrophic forgetting in gradient-based neural networks
Original
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio · 2013
Cited alongside, same era.
Maxout networks
I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. C. Courville, and Y. Bengio · 2013
Cited alongside, same era.
From n to n+1: Multiclass transfer incremental learning
I. Kuzborskij, F. Orabona, and B. Caputo · 2013
Cited alongside, same era.
Distance-based image classification: Generalizing to new classes at near-zero cost
T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka · 2013
Cited alongside, same era.
Compete to compute
R. K. Srivastava, J. Masci, S. Kazerounian, F. Gomez, and J. Schmidhuber · 2013
Cited alongside, same era.
Generative adversarial nets
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio · 2014
Cited alongside, same era.
Incremental learning of NCM forests for large-scale image classification
M. Ristin, M. Guillaumin, J. Gall, and L. Van Gool · 2014
Cited alongside, same era.
Fitnets: Hints for thin deep nets
Original
A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio · 2014
Cited alongside, same era.
Dropout: a simple way to prevent neural networks from overfitting
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov · 2014
Cited alongside, same era.
Mnist handwritten digit database
Y. LeCun, C. Cortes, and C. Burges
Cited in the paper.

Then

Error-driven incremental learning in deep convolutional neural network for large-scale image classification
T. Xiao, J. Zhang, K. Yang, Y. Peng, and Z. Zhang · 2014
Later among the works it cites.
Distilling the knowledge in a neural network
Original
G. Hinton, O. Vinyals, and J. Dean · 2015
Later among the works it cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
S. Ioffe and C. Szegedy · 2015
Later among the works it cites.
Active long term memory networks
Original
T. Furlanello, J. Zhao, A. M. Saxe, L. Itti, and B. S. Tjan · 2016
Later among the works it cites.
Less-forgetting learning in deep neural networks
Original
H. Jung, J. Ju, M. Jung, and J. Kim · 2016
Later among the works it cites.
Learning without forgetting
Z. Li and D. Hoiem · 2016
Later among the works it cites.
Neural dataset generality
Original
R. Venkatesan, V. Gattupalli, and B. Li · 2016
Later among the works it cites.
Diving deeper into mentee networks
Original
R. Venkatesan and B. Li · 2016
Later among the works it cites.
Overcoming catastrophic forgetting in neural networks
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell · 2017
Closest in time.
iCaRL: Incremental classifier and representation learning
S.-A. Rebuffi, A. Kolesnikov, and C. H. Lampert · 2017
Closest in time.
Continual learning in generative adversarial nets
Original
A. Seff, A. Beatson, D. Suo, and H. Liu · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…