2020

Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting

Xie, Zeke, He, Fengxiang, Fu, Shaopeng et al.

Understand

Deep learning is often criticized by two serious issues which rarely exist in natural nervous systems: overfitting and catastrophic forgetting.

It can even memorize randomly labelled data, which has little knowledge behind the instance-label pairs.
When a deep network continually learns over time by accommodating new tasks, it usually quickly overwrites the knowledge learned from previous tasks.
Referred to as the {\it neural variability}, it is well-known in neuroscience that human brain reactions exhibit substantial variability even in response to the same stimulus.

Built on

Where is the information in a deep neural network?
Original
Achille, A., Paolini, G., and Soatto, S. (2019) · 1905
Earlier work this paper cites.
Emergence of invariance and disentanglement in deep representations
Achille, A. and Soatto, S. (2018) · 1980
Earlier work this paper cites.
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · 1989
Earlier work this paper cites.
On the approximate realization of continuous mappings by neural networks
Funahashi, K.-I. (1989) · 1989
Earlier work this paper cites.
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., White, H., et al. (1989) · 1989
Earlier work this paper cites.
Catastrophic interference in connectionist networks: The sequential learning problem
McCloskey, M. and Cohen, N. J. (1989) · 1989
Earlier work this paper cites.
Keeping the neural networks simple by minimizing the description length of the weights
Hinton, G. E. and Van Camp, D. (1993) · 1993
Earlier work this paper cites.
Some new results on neural network approximation
Hornik, K. (1993) · 1993
Earlier work this paper cites.
The effects of adding noise during backpropagation training on a generalization performance
An, G. (1996) · 1996
Earlier work this paper cites.
Flat minima
Hochreiter, S. and Schmidhuber, J. (1997) · 1997
Earlier work this paper cites.
Online learning and stochastic approximations
Bottou, L. (1998) · 1998
Earlier work this paper cites.
The mnist database of handwritten digits
LeCun, Y. (1998) · 1998
Earlier work this paper cites.
Robustness of bayesian neural networks to gradient-based attacks
Original
Carbone, G., Wicker, M., Laurenti, L., Patane, A., Bortolussi, L., and Sanguinetti, G. (2020) · 2002
Earlier work this paper cites.
Improving generalization by controlling label-noise information in neural network weights
Original
Harutyunyan, H., Reing, K., Steeg, G. V., and Galstyan, A. (2020) · 2002
Earlier work this paper cites.
Insights into the ageing mind: a view from cognitive neuroscience
Hedden, T. and Gabrieli, J. D. (2004) · 2004
Earlier work this paper cites.
Neuronal variability: noise or part of the signal?
Stein, R. B., Gossen, E. R., and Jones, K. E. (2005) · 2005
Earlier work this paper cites.
Neural variability in premotor cortex provides a signature of motor preparation
Churchland, M. M., Byron, M. Y., Ryu, S. I., Santhanam, G., and Shenoy, K. V. (2006) · 2006
Earlier work this paper cites.
Adai: Separating the effects of adaptive learning rate and momentum inertia
Original
Xie, Z., Wang, X., Zhang, H., Sato, I., and Sugiyama, M. (2020b) · 2006
Earlier work this paper cites.
Performance variability enables adaptive plasticity of ‘crystallized’adult birdsong
Tumer, E. C. and Brainard, M. S. (2007) · 2007
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A., Hinton, G., et al. (2009) · 2009
Earlier work this paper cites.
Stimulus onset quenches neural variability: a widespread cortical phenomenon
Churchland, M. M., Byron, M. Y., Cunningham, J. P., Sugrue, L. P., Cohen, M. R., Corrado, G. S., Newsome, W. T., Clark, A. M., Hosseini, P., Scott, B. B., et al. (2010) · 2010
Earlier work this paper cites.
Perspective on variability in the development of human action
Fetters, L. (2010) · 2010
Earlier work this paper cites.
Practical variational inference for neural networks
Graves, A. (2011) · 2011
Earlier work this paper cites.

Similar

Changes in the neural control of a complex motor sequence during learning
Ölveczky, B. P., Otchy, T. M., Goldberg, J. H., Aronov, D., and Fee, M. S. (2011) · 2011
Cited alongside, same era.
Bayesian learning via stochastic gradient langevin dynamics
Welling, M. and Teh, Y. W. (2011) · 2011
Cited alongside, same era.
Stable weight decay regularization
Original
Xie, Z., Sato, I., and Sugiyama, M. (2020a) · 2011
Cited alongside, same era.
Elements of information theory
Cover, T. M. and Thomas, J. A. (2012) · 2012
Cited alongside, same era.
Randomized smoothing for stochastic optimization
Duchi, J. C., Bartlett, P. L., and Wainwright, M. J. (2012) · 2012
Cited alongside, same era.
An empirical investigation of catastrophic forgetting in gradient-based neural networks
Original
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. (2013) · 2013
Cited alongside, same era.
Impact of parkinson’s disease and dopaminergic medication on adaptation to explicit and implicit visuomotor perturbations
Mongeon, D., Blanchet, P., and Messier, J. (2013) · 2013
Cited alongside, same era.
On the importance of initialization and momentum in deep learning
Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013) · 2013
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J. (2014) · 2014
Cited alongside, same era.
Weight uncertainty in neural networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. (2015) · 2015
Cited alongside, same era.
Neural variability: friend or foe?
Dinstein, I., Heeger, D. J., and Behrmann, M. (2015) · 2015
Cited alongside, same era.
Deep learning
LeCun, Y., Bengio, Y., and Hinton, G. (2015) · 2015
Cited alongside, same era.

Then

Random gradient-free minimization of convex functions
Nesterov, Y. and Spokoiny, V. (2017) · 2017
Later among the works it cites.
Exploring generalization in deep learning
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017) · 2017
Later among the works it cites.
Information-theoretic analysis of generalization capability of learning algorithms
Xu, A. and Raginsky, M. (2017) · 2017
Later among the works it cites.
Continual learning through synaptic intelligence
Zenke, F., Poole, B., and Ganguli, S. (2017) · 2017
Later among the works it cites.
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017) · 2017
Later among the works it cites.
Memory aware synapses: Learning what (not) to forget
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. (2018) · 2018
Later among the works it cites.
On the optimization of deep networks: Implicit acceleration by overparameterization
Arora, S., Cohen, N., and Hazan, E. (2018) · 2018
Later among the works it cites.
Co-teaching: Robust training of deep neural networks with extremely noisy labels
Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., and Sugiyama, M. (2018) · 2018
Later among the works it cites.
Fast and scalable bayesian deep learning by weight-perturbation in adam
Khan, M., Nielsen, D., Tangkaratt, V., Lin, W., Gal, Y., and Srivastava, A. (2018) · 2018
Later among the works it cites.
Learning overparameterized neural networks via stochastic gradient descent on structured data
Li, Y. and Liang, Y. (2018) · 2018
Later among the works it cites.
Decoupled weight decay regularization
Loshchilov, I. and Hutter, F. (2018) · 2018
Later among the works it cites.
Mobilenetv2: Inverted residuals and linear bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. (2018) · 2018
Later among the works it cites.
Smoothout: Smoothing out sharp minima to improve generalization in deep learning
Original
Wen, W., Wang, Y., Yan, F., Xu, C., Wu, C., Chen, Y., and Li, H. (2018) · 2018
Later among the works it cites.
Improving the antinoise ability of dnns via a bio-inspired noise adaptive activation function rand softplus
Chen, Y., Mai, Y., Xiao, J., and Zhang, L. (2019) · 2019
Later among the works it cites.
Control batch size and learning rate to generalize well: Theoretical and empirical evidence
He, F., Liu, T., and Tao, D. (2019) · 2019
Later among the works it cites.
Calculating the mutual information between two spike trains
Houghton, C. (2019) · 2019
Later among the works it cites.
Effect of depth and width on local minima in deep learning
Kawaguchi, K., Huang, J., and Kaelbling, L. P. (2019) · 2019
Later among the works it cites.
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. (2019) · 2019
Later among the works it cites.
Toward understanding the importance of noise in training neural networks
Zhou, M., Liu, T., Li, Y., Lin, D., Zhou, E., and Zhao, T. (2019) · 2019
Later among the works it cites.
The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects
Zhu, Z., Wu, J., Yu, B., Wu, L., and Ma, J. (2019) · 2019
Later among the works it cites.
Being bayesian, even just a bit, fixes overconfidence in relu networks
Kristiadi, A., Hein, M., and Hennig, P. (2020) · 2020
Closest in time.
A theoretical analysis of catastrophic forgetting through the ntk overlap matrix
Doan, T., Bennani, M. A., Mazoure, B., Rabusseau, G., and Alquier, P. (2021) · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…