2017

Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science

Mocanu, Decebal Constantin, Mocanu, Elena, Stone, Peter et al.

Understand

Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods.

Taking inspiration from the network properties of biological neural networks (e.g.
sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers.
Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erd\H{o}s-R\'enyi random graph) of two consecutive layers of neurons into a scale-free topology, during learning.

Built on

On random graphs i
Erdős, P. & Rényi, A · 1959
Earlier work this paper cites.
Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms (Spartan, 1962)
Rosenblatt, F · 1962
Earlier work this paper cites.
Information processing in dynamical systems: Foundations of harmony theory
Smolensky, P · 1986
Earlier work this paper cites.
Learning representations by back-propagating errors
Rumelhart, D., Hintont, G. & Williams, R · 1986
Earlier work this paper cites.
Approximation by superpositions of a sigmoidal function
Cybenko, G · 1989
Earlier work this paper cites.
Generalization by weight-elimination with application to forecasting
Weigend, A. S., Rumelhart, D. E. & Huberman, B. A · 1991
Earlier work this paper cites.
Evolving neural network connectivity
McDonnell, J. R. & Waagen, D · 1993
Earlier work this paper cites.
Collective dynamics of ’small-world’ networks
Watts, D. J. & Strogatz, S. H · 1998
Earlier work this paper cites.
Introduction to Reinforcement Learning (MIT Press, Cambridge, MA, USA, 1998)
Sutton, R. S. & Barto, A. G · 1998
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P · 1998
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P · 1998
Earlier work this paper cites.
Emergence of scaling in random networks
Barabási, A.-L. & Albert, R · 1999
Earlier work this paper cites.
Exploring complex networks
Strogatz, S. H · 2001
Earlier work this paper cites.
Random graphs with arbitrary degree distributions and their applications
Newman, M. E., Strogatz, S. H. & Watts, D. J · 2001
Earlier work this paper cites.
The Elements of Statistical Learning (Springer New York Inc., New York, NY, USA, 2001)
Hastie, T., Tibshirani, R. & Friedman, J · 2001
Earlier work this paper cites.
Evolving neural networks through augmenting topologies
Stanley, K. O. & Miikkulainen, R · 2002
Earlier work this paper cites.
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E · 2002
Earlier work this paper cites.
The Cambridge dictionary of statistics (Cambridge University Press, Cambridge, UK; New York, 2002)
Everitt, B · 2002
Earlier work this paper cites.
Statistical mechanics of complex networks
Albert, R. & Barabási, A.-L · 2002
Earlier work this paper cites.
Evolutionary function approximation for reinforcement learning
Whiteson, S. & Stone, P · 2006
Earlier work this paper cites.
Reducing the Dimensionality of Data with Neural Networks
Hinton, G. E. & Salakhutdinov, R. R · 2006
Earlier work this paper cites.
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S. & Teh, Y.-W · 2006
Earlier work this paper cites.
Functional census of mutation sequence spaces: the example of p53 cancer rescue mutants
Danziger, S. A. et al · 2006
Earlier work this paper cites.

Similar

Pattern Recognition and Machine Learning (Information Science and Statistics) (Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006)
Bishop, C. M · 2006
Cited alongside, same era.
The rate adapting poisson model for information retrieval and object recognition
Gehler, P. V., Holub, A. D. & Welling, M · 2006
Cited alongside, same era.
Restricted boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A. & Hinton, G · 2007
Cited alongside, same era.
On the quantitative analysis of deep belief networks
Salakhutdinov, R. & Murray, I · 2008
Cited alongside, same era.
Classification using discriminative restricted boltzmann machines
Larochelle, H. & Bengio, Y · 2008
Cited alongside, same era.
The tradeoffs of large scale learning
Bottou, L. & Bousquet, O · 2008
Cited alongside, same era.
Complex brain networks: graph theoretical analysis of structural and functional systems
Bullmore, E. & Sporns, O · 2009
Cited alongside, same era.
Learning deep architectures for ai
Bengio, Y · 2009
Cited alongside, same era.
Power-law distributions in empirical data
Clauset, A., Shalizi, C. R. & Newman, M. E. J · 2009
Cited alongside, same era.
A novel connectionist system for unconstrained handwriting recognition
Graves, A. et al · 2009
Cited alongside, same era.
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A · 2009
Cited alongside, same era.
Rectified linear units improve restricted boltzmann machines
Nair, V. & Hinton, G. E · 2010
Cited alongside, same era.

Then

Scientific method: Statistical errors
Nuzzo, R · 2014
Later among the works it cites.
Human-level control through deep reinforcement learning
Mnih, V. et al · 2015
Later among the works it cites.
Deep learning
LeCun, Y., Bengio, Y. & Hinton, G · 2015
Later among the works it cites.
Learning both weights and connections for efficient neural network
Han, S., Pool, J., Tran, J. & Dally, W · 2015
Later among the works it cites.
No-reference video quality measurement: added value of machine learning
Mocanu, D. C. et al · 2015
Later among the works it cites.
How far can we go without convolution: Improving fully-connected networks
Original
Lin, Z., Memisevic, R. & Konda, K · 2015
Later among the works it cites.
On the synergy of network science and artificial intelligence
Mocanu, D. C · 2016
Later among the works it cites.
A topological insight into restricted boltzmann machines
Mocanu, D. C., Mocanu, E., Nguyen, P. H., Gibescu, M. & Liotta, A · 2016
Later among the works it cites.
Neural networks with differentiable structure
Original
Miconi, T · 2016
Later among the works it cites.
Theano: A Python framework for fast computation of mathematical expressions
Original
Al-Rfou, R., Alain, G., Almahairi, A. & et al., C. A · 2016
Later among the works it cites.
Do deep convolutional nets really need to be deep and convolutional?
Urban, G. et al · 2016
Later among the works it cites.
Deep learning with s-shaped rectified linear activation units
Jin, X. et al · 2016
Later among the works it cites.
Network science (Cambridge University Press, 2016)
Barabási, A.-L · 2016
Later among the works it cites.
Mastering the game of go with deep neural networks and tree search
Silver, D. et al · 2016
Later among the works it cites.
Fast convnets using group-wise brain damage
Lebedev, V. & Lempitsky, V · 2016
Later among the works it cites.
Evolving deep neural networks
Original
Miikkulainen, R. et al · 2017
Closest in time.
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
Original
Salimans, T., Ho, J., Chen, X., Sidor, S. & Openai, I. S · 2017
Closest in time.
Homer1a drives homeostatic scaling-down of excitatory synapses during sleep
Diering, G. H. et al · 2017
Closest in time.
Ultrastructural evidence for synaptic scaling across the wake/sleep cycle
de Vivo, L. et al · 2017
Closest in time.
The power of sparsity in convolutional neural networks
Original
Changpinyo, S., Sandler, M. & Zhmoginov, A · 2017
Closest in time.
Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
Original
Xiao, H., Rasul, K. & Vollgraf, R · 2017
Closest in time.
Deep Neuroevolution: Genetic Algorithms are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
Original
Such, F. P. et al · 2018
Closest in time.
Decentralized dynamic understanding of hidden relations in complex networks
Mocanu, D. C., Exarchakos, G. & Liotta, A · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…