2017

Channel Pruning for Accelerating Very Deep Neural Networks

He, Yihui, Zhang, Xiangyu, Sun, Jian

Understand

In this paper, we introduce a new channel pruning method to accelerate very deep convolutional neural networks.Given a trained CNN model, we propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction.

We further generalize this algorithm to multi-layer and multi-branch cases.
Our method reduces the accumulated error and enhance the compatibility with various architectures.
Our pruned VGG-16 achieves the state-of-the-art results by 5x speed-up along with only 0.3% increase of error.

Built on

Better subset regression using the nonnegative garrote
L. Breiman · 1995
Earlier work this paper cites.
Regression shrinkage and selection via the lasso
R. Tibshirani · 1996
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner · 1998
Earlier work this paper cites.
The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman · 2007
Earlier work this paper cites.
Scalable parallel programming with CUDA
J. Nickolls, I. Buck, M. Garland, and K. Skadron · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky and G. Hinton · 2009
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
V. Nair and G. E. Hinton · 2010
Earlier work this paper cites.
Scikit-learn: Machine learning in Python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Fast training of convolutional networks through ffts
Original
M. Mathieu, M. Henaff, and Y. LeCun · 2013
Earlier work this paper cites.
Restructuring of deep neural network acoustic models with singular value decomposition
J. Xue, J. Li, and Y. Gong · 2013
Earlier work this paper cites.
cudnn: Efficient primitives for deep learning
Original
S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer · 2014
Earlier work this paper cites.
Exploiting linear structure within convolutional networks for efficient evaluation
E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus · 2014
Earlier work this paper cites.
Compressing deep convolutional networks using vector quantization
Original
Y. Gong, L. Liu, M. Yang, and L. Bourdev · 2014
Earlier work this paper cites.
Speeding up convolutional neural networks with low rank expansions
Original
M. Jaderberg, A. Vedaldi, and A. Zisserman · 2014
Earlier work this paper cites.
Caffe: Convolutional architecture for fast feature embedding
Original
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell · 2014
Earlier work this paper cites.
Speeding-up convolutional neural networks using fine-tuned cp-decomposition
Original
V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky · 2014
Earlier work this paper cites.

Similar

Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Original
K. Simonyan and A. Zisserman · 2014
Cited alongside, same era.
Fast convolutional nets with fbfft: A gpu performance evaluation
Original
N. Vasilache, J. Johnson, M. Mathieu, S. Chintala, S. Piantino, and Y. LeCun · 2014
Cited alongside, same era.
Structured pruning of deep convolutional neural networks
Original
S. Anwar, K. Hwang, and W. Sung · 2015
Cited alongside, same era.
Fast r-cnn
R. Girshick · 2015
Cited alongside, same era.
Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding
Original
S. Han, H. Mao, and W. J. Dally · 2015
Cited alongside, same era.
Learning both weights and connections for efficient neural network
S. Han, J. Pool, J. Tran, and W. Dally · 2015
Cited alongside, same era.
Deep residual learning for image recognition
Original
K. He, X. Zhang, S. Ren, and J. Sun · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
S. Ioffe and C. Szegedy · 2015
Cited alongside, same era.
Compression of deep convolutional neural networks for fast and low power mobile applications
Original
Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin · 2015
Cited alongside, same era.
Fast algorithms for convolutional neural networks
Original
A. Lavin · 2015
Cited alongside, same era.
Fast convnets using group-wise brain damage
Original
V. Lebedev and V. Lempitsky · 2015
Cited alongside, same era.

Then

Data-free parameter pruning for deep neural networks
Original
S. Srinivas and R. V. Babu · 2015
Later among the works it cites.
Going deeper with convolutions
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich · 2015
Later among the works it cites.
Learning the number of neurons in deep networks
J. M. Alvarez and M. Salzmann · 2016
Later among the works it cites.
Compact deep convolutional neural networks with coarse pruning
Original
S. Anwar and W. Sung · 2016
Later among the works it cites.
Lcnn: Lookup-based convolutional neural network
Original
H. Bagherinezhad, M. Rastegari, and A. Farhadi · 2016
Later among the works it cites.
Xception: Deep learning with depthwise separable convolutions
Original
F. Chollet · 2016
Later among the works it cites.
Binarynet: Training deep neural networks with weights and activations constrained to+ 1 or-1
Original
M. Courbariaux and Y. Bengio · 2016
Later among the works it cites.
Dynamic network surgery for efficient dnns
Y. Guo, A. Yao, and Y. Chen · 2016
Later among the works it cites.
Eie: efficient inference engine on compressed deep neural network
S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally · 2016
Later among the works it cites.
Network trimming: A data-driven neuron pruning approach towards efficient deep architectures
Original
H. Hu, R. Peng, Y.-W. Tai, and C.-K. Tang · 2016
Later among the works it cites.
Speed/accuracy trade-offs for modern convolutional object detectors
Original
J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, et al · 2016
Later among the works it cites.
Pruning filters for efficient convnets
Original
H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf · 2016
Later among the works it cites.
Xnor-net: Imagenet classification using binary convolutional neural networks
M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi · 2016
Later among the works it cites.
Learning structured sparsity in deep neural networks
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li · 2016
Later among the works it cites.
Aggregated residual transformations for deep neural networks
Original
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He · 2016
Later among the works it cites.
Designing energy-efficient convolutional neural networks using energy-aware pruning
Original
T.-J. Yang, Y.-H. Chen, and V. Sze · 2016
Later among the works it cites.
Accelerating very deep convolutional networks for classification and detection
X. Zhang, J. Zou, K. He, and J. Sun · 2016
Later among the works it cites.
Less is more: Towards compact cnns
H. Zhou, J. M. Alvarez, and F. Porikli · 2016
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…