2020

Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning

Park, Sejun, Lee, Jaeho, Mo, Sangwoo et al.

Understand

Magnitude-based pruning is one of the simplest methods for pruning neural networks.

Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable performances for pruning modern architectures.
Based on the observation that magnitude-based pruning indeed minimizes the Frobenius distortion of a linear operator corresponding to a single layer, we develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization.
Our experimental results demonstrate that the proposed method consistently outperforms magnitude-based pruning on various networks, including VGG and ResNet, particularly in the high-sparsity regime.

Built on

Some NP-complete problems in quadratic and nonlinear programming
K. G. Murty and S. N. Kabadi · 1987
Earlier work this paper cites.
Optimal brain damage
Y. LeCun, J. S. Denker, and S. A. Solla · 1989
Earlier work this paper cites.
Second order derivatives for network pruning: Optimal brain surgeon
B. Hassibi and D. G. Stork · 1993
Earlier work this paper cites.
Bayesian learning for neural networks
R. M. Neal · 1996
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner · 1998
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky and G. Hinton · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
X. Glorot and Y. Bengio · 2010
Earlier work this paper cites.
Learning both weights and connections for efficient neural networks
S. Han, J. Pool, J. Tran, and W. J. Dally · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
S. Ioffe and C. Szegedy · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
D. P. Kingma and J. Ba · 2015
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
K. Simonyan and A. Zisserman · 2015
Earlier work this paper cites.
Deep Learning
I. Goodfellow, Y. Bengio, and A. Courville · 2016
Earlier work this paper cites.

Similar

Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Learning structured sparsity in deep networks
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li · 2016
Cited alongside, same era.
Wide residual networks
S. Zagoruyko and N. Komodakis · 2016
Cited alongside, same era.
Learning to prune deep neural networks via layer-wise optimal brain surgeon
Xin Dong, Shangyu Chen, and Sinno Pan · 2017
Cited alongside, same era.
Runtime neural pruning
J. Lin, Y. Rao, J. Lu, and J. Zhou · 2017
Cited alongside, same era.
Learning efficient convolutional networks through network slimming
Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang · 2017
Cited alongside, same era.
Bayesian compression for deep learning
C. Louizos, K. Ullrich, and M. Welling · 2017
Cited alongside, same era.
Variational dropout sparsified deep neural networks
D. Molchanov, A. Ashukha, and D. Vetrov · 2017
Cited alongside, same era.
To prune, or not to prune: exploring the efficacy of pruning for model compression
Original
M. Zhu and S. Gupta · 2017
Cited alongside, same era.
On the optimization of deep networks: Implicit acceleration by overparametrization
S. Arora, N. Cohen, and E. Hazan · 2018
Cited alongside, same era.
Deep rewiring: training very sparse deep networks
G. Bellec, D. Kappel, W. Maass, and R. Legenstein · 2018
Cited alongside, same era.
Compressing neural networks using the variational information bottleneck
B. Dai, C. Zhu, B. Guo, and D. Wipf · 2018
Cited alongside, same era.

Then

Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced
S. S. Du, W. Hu, and J. D. Lee · 2018
Later among the works it cites.
AMC: AutoML for model compression and acceleration on mobile devices
Y. He, J. Lin, Z. Liu, H. Wang, L. Li, and S. Han · 2018
Later among the works it cites.
Learning sparse neural networks through l 0 l_{0} regularization
C. Louizos, M. Welling, and D. P. Kingma · 2018
Later among the works it cites.
MobileNetV2: Inverted residuals and linear bottlenecks
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen · 2018
Later among the works it cites.
Rethinking smaller-norm-less-informative assumption in channel pruning of convolutional layers
J. Ye, X. Lu, Z. Lin, and J. Z. Wang · 2018
Later among the works it cites.
Learning to share: simultaneous parameter tying and sparsification in deep learning
D. Zhang, H. Wang, M. Figueiredo, and L. Balzano · 2018
Later among the works it cites.
Centripetal SGD for pruning very deep convolutional networks with complicated structure
X. Ding, G. Ding, Y. Guo, and J. Han · 2019
Later among the works it cites.
The lottery ticket hypothesis: Finding sparse, trainable neural networks
J. Frankle and M. Carbin · 2019
Later among the works it cites.
The state of sparsity in deep neural networks
T. Gale, E. Elsen, and S. Hooker · 2019
Later among the works it cites.
Rethinking the value of network pruning
Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell · 2019
Later among the works it cites.
The singular values of convolutional layers
H. Sedghi, V. Gupta, and P. M. Long · 2019
Later among the works it cites.
BackPACK: Packing more into backprop
F. Dangel, F. Kunstner, and P. Hennig · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…