2022

When, where, and how to add new neurons to ANNs

Maile, Kaitlin, Rachelson, Emmanuel, Luga, Hervé et al.

Understand

Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of structural learning like pruning.

By decomposing it into triggers and initializations, we introduce a framework for studying the various facets of neurogenesis: when, where, and how to add neurons during the learning process.
We present the Neural Orthogonality (NORTH*) suite of neurogenesis strategies, combining layer-wise triggers and initializations based on the orthogonality of activations or weights to dynamically grow performant networks that converge to an efficient size.
We evaluate our contributions against other recent neurogenesis works across a variety of supervised learning tasks.

Built on

Quantifying the carbon emissions of machine learning
Original
Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. (2019) · 1910
Earlier work this paper cites.
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986) · 1986
Earlier work this paper cites.
Dynamic node creation in backpropagation networks
Ash, T. (1989) · 1989
Earlier work this paper cites.
Nonlinear programming
Bertsekas, D. P. (1997) · 1997
Earlier work this paper cites.
Convex neural networks
Bengio, Y., Roux, N., Vincent, P., Delalleau, O., and Marcotte, P. (2005) · 2005
Earlier work this paper cites.
A new constructive algorithm for architectural and functional adaptation of artificial neural networks
Islam, M. M., Sattar, M. A., Amin, M. F., Yao, X., and Murase, K. (2009) · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Glorot, X. and Bengio, Y. (2010) · 2010
Earlier work this paper cites.
The MNIST database of handwritten digit images for machine learning research
Deng, L. (2012) · 2012
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, A. M., McClelland, J. L., and Ganguli, S. (2014) · 2014
Earlier work this paper cites.
ADAM: A method for stochastic optimization
Kingma, D. P. and Ba, J. (2015) · 2015
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Simonyan, K. and Zisserman, A. (2015) · 2015
Earlier work this paper cites.

Similar

Net2Net: Accelerating learning via knowledge transfer
Chen, T., Goodfellow, I., and Shlens, J. (2016) · 2016
Cited alongside, same era.
Deep Learning
Goodfellow, I., Bengio, Y., and Courville, A. (2016) · 2016
Cited alongside, same era.
Network morphism
Wei, T., Wang, C., Rui, Y., and Chen, C. W. (2016) · 2016
Cited alongside, same era.
Pruning filters for efficient convnets
Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P. (2017) · 2017
Cited alongside, same era.
Escaping flat areas via function-preserving structural network modifications
Kilcher, Y., Bécigneul, G., and Hofmann, T. (2018) · 2018
Cited alongside, same era.
NeST: A neural network synthesis tool based on a grow-and-prune paradigm
Dai, X., Yin, H., and Jha, N. K. (2019) · 2019
Cited alongside, same era.
Efficient network construction through structural plasticity
Du, X., Li, Z., Ma, Y., and Cao, Y. (2019) · 2019
Cited alongside, same era.
Neural architecture search: A survey
Elsken, T., Metzen, J. H., and Hutter, F. (2019) · 2019
Cited alongside, same era.
Splitting steepest descent for growing neural architectures
Wu, L., Wang, D., and Liu, Q. (2019) · 2019
Cited alongside, same era.
Model compression and hardware acceleration for neural networks: A comprehensive survey
Deng, L., Li, G., Han, S., Shi, L., and Xie, Y. (2020) · 2020
Cited alongside, same era.
FBNetV2: Differentiable neural architecture search for spatial and channel dimensions
Wan, A., Dai, X., Zhang, P., He, Z., Tian, Y., Xie, S., Wu, B., Yu, M., Xu, T., Chen, K., et al. (2020) · 2020
Cited alongside, same era.

Then

AutoGrow: Automatic layer growing in deep convolutional networks
Wen, W., Yan, F., Chen, Y., and Li, H. (2020) · 2020
Later among the works it cites.
Firefly neural architecture descent: a general approach for growing neural networks
Wu, L., Liu, B., Stone, P., and Liu, Q. (2020) · 2020
Later among the works it cites.
Numerical influence of ReLU’(0) on backpropagation
Bertoin, D., Bolte, J., Gerchinovitz, S., and Pauwels, E. (2021) · 2021
Later among the works it cites.
Batch normalization orthogonalizes representations in deep random networks
Daneshmand, H., Joudaki, A., and Bach, F. (2021) · 2021
Later among the works it cites.
Implicit under-parameterization inhibits data-efficient deep reinforcement learning
Kumar, A., Agarwal, R., Ghosh, D., and Levine, S. (2021) · 2021
Later among the works it cites.
Green algorithms: Quantifying the carbon footprint of computation
Lannelongue, L., Grealey, J., and Inouye, M. (2021) · 2021
Later among the works it cites.
Neural architecture search without training
Mellor, J., Turner, J., Storkey, A., and Crowley, E. J. (2021) · 2021
Later among the works it cites.
Wide residual networks
Original
Zagoruyko, S. and Komodakis, N. (2016) · 2021
Later among the works it cites.
GradMax: Growing neural networks using gradient information
Evci, U., van Merriënboer, B., Unterthiner, T., Pedregosa, F., and Vladymyrov, M. (2022) · 2022
Closest in time.
Understanding dimensional collapse in contrastive self-supervised learning
Jing, L., Vincent, P., LeCun, Y., and Tian, Y. (2022) · 2022
Closest in time.
Understanding and preventing capacity loss in reinforcement learning
Lyle, C., Rowland, M., and Dabney, W. (2022) · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…