2021

Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization

Tessera, Kale-ab, Hooker, Sara, Rosman, Benjamin

Understand

Training sparse networks to converge to the same performance as dense neural architectures has proven to be elusive.

Recent work suggests that initialization is the key.
However, while this direction of research has had some success, focusing on initialization alone appears to be inadequate.
In this paper, we take a broader view of training sparse networks and consider the role of regularization, optimization, and architecture choices on sparse models.

Built on

Stabilizing the lottery ticket hypothesis
Original
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin · 1903
Earlier work this paper cites.
Linear mode connectivity and the lottery ticket hypothesis
Original
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin · 1912
Earlier work this paper cites.
A new measure of rank correlation
Maurice G Kendall · 1938
Earlier work this paper cites.
Individual comparisons by ranking methods
Frank Wilcoxon · 1945
Earlier work this paper cites.
A stochastic approximation method
Herbert Robbins and Sutton Monro · 1951
Earlier work this paper cites.
Some methods of speeding up the convergence of iteration methods
Boris T Polyak · 1964
Earlier work this paper cites.
On the approximate realization of continuous mappings by neural networks
Ken-Ichi Funahashi · 1989
Earlier work this paper cites.
Comparing biases for minimal network construction with back-propagation
Stephen José Hanson and Lorien Y Pratt · 1989
Earlier work this paper cites.
Multilayer feedforward networks are universal approximators
Kurt Hornik, Maxwell Stinchcombe, Halbert White, et al · 1989
Earlier work this paper cites.
Optimal Brain Damage
Yann LeCun, John S. Denker, and Sara A. Solla · 1989
Earlier work this paper cites.
Skeletonization: A technique for trimming the fat from a network via relevance assessment
Michael C Mozer and Paul Smolensky · 1989
Earlier work this paper cites.
Optimal brain damage
Yann Le Cun, John S. Denker, and Sara A. Solla · 1990
Earlier work this paper cites.
A simple procedure for pruning back-propagation trained neural networks
Ehud D Karnin · 1990
Earlier work this paper cites.
Untersuchungen zu dynamischen neuronalen netzen
Sepp Hochreiter · 1991
Earlier work this paper cites.
A simple weight decay can improve generalization
Anders Krogh and John A Hertz · 1992
Earlier work this paper cites.
Connectionist learning of belief networks
Radford M Neal · 1992
Earlier work this paper cites.
Optimal brain surgeon and general network pruning
B. Hassibi, D. G. Stork, and G. J. Wolff · 1993
Earlier work this paper cites.
Learning long-term dependencies with gradient descent is difficult
Yoshua Bengio, Patrice Simard, and Paolo Frasconi · 1994
Earlier work this paper cites.
Regression shrinkage and selection via the lasso
Robert Tibshirani · 1996
Earlier work this paper cites.
Sparse connection and pruning in large dynamic artificial neural networks, 1997
Nikko Ström · 1997
Earlier work this paper cites.
Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al · 2001
Earlier work this paper cites.
On the behavior of the gradient norm in the steepest descent method
Jorge Nocedal, Annick Sartenaer, and Ciyou Zhu · 2002
Earlier work this paper cites.
Statistical comparisons of classifiers over multiple data sets
Janez Demšar · 2006
Earlier work this paper cites.
Cifar-10 and cifar-100 datasets
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton · 2009
Earlier work this paper cites.
Handbook of biological statistics , volume 2
John H McDonald · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton · 2010
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
John Duchi, Elad Hazan, and Yoram Singer · 2011
Earlier work this paper cites.
Deep sparse rectifier neural networks
Xavier Glorot, Antoine Bordes, and Yoshua Bengio · 2011
Earlier work this paper cites.
Neural networks for machine learning lecture 6a overview of mini-batch gradient descent
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Adadelta: an adaptive learning rate method
Original
Matthew D Zeiler · 2012
Earlier work this paper cites.
On the difficulty of training recurrent neural networks
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio · 2013
Earlier work this paper cites.

Similar

On the importance of initialization and momentum in deep learning
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton · 2013
Cited alongside, same era.
1.1 computing’s energy problem (and what we can do about it)
M. Horowitz · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Cited alongside, same era.
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Cited alongside, same era.
Fast and accurate deep network learning by exponential linear units (elus)
Original
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter · 2015
Cited alongside, same era.
Learning both weights and connections for efficient neural networks
Song Han, Jeff Pool, John Tran, and William J. Dally · 2015
Cited alongside, same era.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Sergey Ioffe and Christian Szegedy · 2015
Cited alongside, same era.
Deep learning with s-shaped rectified linear activation units
Original
Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, and Shuicheng Yan · 2015
Cited alongside, same era.
Highway networks
Original
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber · 2015
Cited alongside, same era.
Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Cited alongside, same era.
The power of depth for feedforward neural networks
Ronen Eldan and Ohad Shamir · 2016
Cited alongside, same era.

Then

Snip: Single-shot network pruning based on connection sensitivity
Original
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr · 2018
Later among the works it cites.
Rethinking the value of network pruning
Original
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell · 2018
Later among the works it cites.
Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta · 2018
Later among the works it cites.
Evolutionary pruning of transfer learned deep convolutional neural network for breast cancer diagnosis in digital breast tomosynthesis
Ravi K Samala, Heang-Ping Chan, Lubomir M Hadjiiski, Mark A Helvie, Caleb Richter, and Kenny Cha · 2018
Later among the works it cites.
Adaptive methods for nonconvex optimization
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar · 2018
Later among the works it cites.
Bridging the gap between deep learning and sparse matrix format selection
Yue Zhao, Jiajia Li, Chunhua Liao, and Xipeng Shen · 2018
Later among the works it cites.
How can we be so dense? the benefits of using highly sparse representations
Original
Subutai Ahmad and Luiz Scheinkman · 2019
Later among the works it cites.
Sparse networks from scratch: Faster training without losing performance
Original
Tim Dettmers and Luke Zettlemoyer · 2019
Later among the works it cites.
Rigging the Lottery: Making All Tickets Winners
Original
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen · 2019
Later among the works it cites.
The difficulty of training sparse neural networks
Original
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen · 2019
Later among the works it cites.
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Jonathan Frankle and Michael Carbin · 2019
Later among the works it cites.
The State of Sparsity in Deep Neural Networks
Original
Trevor Gale, Erich Elsen, and Sara Hooker · 2019
Later among the works it cites.
What Do Compressed Deep Neural Networks Forget?
Original
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome · 2019
Later among the works it cites.
Fantastic generalization measures and where to find them
Original
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio · 2019
Later among the works it cites.
Characterizing well-behaved vs. pathological deep neural networks
Antoine Labatie · 2019
Later among the works it cites.
A signal propagation perspective for pruning neural networks at initialization
Original
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr · 2019
Later among the works it cites.
Optimizing sparse tensor times matrix on gpus
Yuchen Ma, Jiajia Li, Xiaolong Wu, Chenggang Yan, Jimeng Sun, and Richard Vuduc · 2019
Later among the works it cites.
Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
Original
Hesham Mostafa and Xin Wang · 2019
Later among the works it cites.
Energy and policy considerations for deep learning in nlp, 2019
Emma Strubell, Ananya Ganesh, and Andrew McCallum · 2019
Later among the works it cites.
TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers
P. Warden and D. Situnayake · 2019
Later among the works it cites.
A mean field theory of batch normalization
Original
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz · 2019
Later among the works it cites.
Dive into deep learning
Aston Zhang, Zachary C Lipton, Mu Li, and Alexander J Smola · 2019
Later among the works it cites.
Snap: A 1.67—21.55 tops/w sparse neural acceleration processor for unstructured sparse deep neural network inference in 16nm cmos
Jie-Fang Zhang, Ching-En Lee, Chester Liu, Yakun Sophia Shao, Stephen W Keckler, and Zhengya Zhang · 2019
Later among the works it cites.
Activation function impact on sparse neural networks
Adam Dubowski · 2020
Later among the works it cites.
Gradient flow in sparse neural networks and how lottery tickets win
Original
Utku Evci, Yani A Ioannou, Cem Keskin, and Yann Dauphin · 2020
Later among the works it cites.
Pruning neural networks at initialization: Why are we missing the mark?, 2020
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin · 2020
Later among the works it cites.
Finding trainable sparse networks through neural tangent transfer
Original
Tianlin Liu and Friedemann Zenke · 2020
Later among the works it cites.
Pruning neural networks without any data by iteratively conserving synaptic flow
Original
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli · 2020
Later among the works it cites.
The Computational Limits of Deep Learning
Original
Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso · 2020
Later among the works it cites.
Picking winning tickets before training by preserving gradient flow
Original
Chaoqi Wang, Guodong Zhang, and Roger Grosse · 2020
Later among the works it cites.
Sparch: Efficient architecture for sparse matrix multiplication
Zhekai Zhang, Hanrui Wang, Song Han, and William J Dally · 2020
Later among the works it cites.
A gradient flow framework for analyzing network pruning
Ekdeep Singh Lubana and Robert Dick · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…