2016

Less-forgetting Learning in Deep Neural Networks

Jung, Heechul, Ju, Jeongwoo, Jung, Minju et al.

Understand

A catastrophic forgetting problem makes deep neural networks forget the previously learned information, when learning data collected in new environments, such as by different sensors or in different light conditions.

This paper presents a new method for alleviating the catastrophic forgetting problem.
Unlike previous research, our method does not use any information from the source domain.
Surprisingly, our method is very effective to forget less of the information in the source domain, and we show the effectiveness of our method using several experiments.

Built on

L. Van der Maaten and G. Hinton, “Visualizing data using t-sne,”
2008
Earlier work this paper cites.
S. J. Pan and Q. Yang, “A survey on transfer learning,”
2010
Earlier work this paper cites.
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” 2012, technical Report arXiv:1207.0580
Original
2012
Earlier work this paper cites.
A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in
2013
Earlier work this paper cites.
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio, “An empirical investigation of catastrophic forgeting in gradient-based neural networks,”
Original
2013
Earlier work this paper cites.
R. K. Srivastava, J. Masci, S. Kazerounian, F. Gomez, and J. Schmidhuber, “Compete to compute,” in
2013
Earlier work this paper cites.

Similar

I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, “Maxout networks,” in
2013
Cited alongside, same era.
Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” in
2014
Cited alongside, same era.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”
Original
2014
Cited alongside, same era.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”
Original
2014
Cited alongside, same era.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,”
2014
Cited alongside, same era.

Then

B. Goodrich and I. Arel, “Unsupervised neuron selection for mitigating catastrophic forgetting in neural networks,” in
2014
Later among the works it cites.
B. Goodrich and I. Arel, “Neuron clustering for mitigating catastrophic forgetting in feedforward neural networks,” in
2014
Later among the works it cites.
V. M. Patel, R. Gopalan, R. Li, and R. Chellappa, “Visual domain adaptation: A survey of recent advances,”
2015
Later among the works it cites.
Goodrich and I. Arel, “Mitigating catastrophic forgetting in temporal difference learning with function approximation,” 2015
2015
Later among the works it cites.
T. Lancewicki, B. Goodrich, and I. Arel, “Sequential covariance-matrix estimation with application to mitigating catastrophic forgetting,” in
2015
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…