2020

What is the State of Neural Network Pruning?

Blalock, Davis, Ortiz, Jose Javier Gonzalez, Frankle, Jonathan et al.

Understand

Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years.

We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature.
After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics.
This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades.

Built on

A Signal Propagation Perspective for Pruning Neural Networks at Initialization
Original
Lee, N., Ajanthan, T., Gould, S., and Torr, P. H. S · 1906
Earlier work this paper cites.
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
Original
Morcos, A. S., Yu, H., Paganini, M., and Tian, Y · 1906
Earlier work this paper cites.
Pruning versus clipping in neural networks
Janowsky, S. A · 1989
Earlier work this paper cites.
A simple procedure for pruning back-propagation trained neural networks
Karnin, E. D · 1990
Earlier work this paper cites.
Optimal brain damage
LeCun, Y., Denker, J. S., and Solla, S. A · 1990
Earlier work this paper cites.
Optimal brain surgeon and general network pruning
Hassibi, B., Stork, D. G., and Wolff, G. J · 1993
Earlier work this paper cites.
Pruning algorithms-a survey
Reed, R · 1993
Earlier work this paper cites.
Early brain damage
Tresp, V., Neuneier, R., and Zimmermann, H.-G · 1997
Earlier work this paper cites.
Fast lossless compression of scientific floating-point data
Ratanaworabhan, P., Ke, J., and Burtscher, M · 2006
Earlier work this paper cites.
Super-scalar ram-cpu cache compression
Zukowski, M., Heman, S., Nes, N., and Boncz, P · 2006
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Speeding-up convolutional neural networks using fine-tuned cp-decomposition
Original
Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I., and Lempitsky, V · 2014
Earlier work this paper cites.
Fixed-rate compressed floating-point arrays
Lindstrom, P · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K. and Zisserman, A · 2014
Earlier work this paper cites.
Learning both weights and connections for efficient neural network
Han, S., Pool, J., Tran, J., and Dally, W · 2015
Earlier work this paper cites.
Compression of deep convolutional neural networks for fast and low power mobile applications
Original
Kim, Y.-D., Park, E., Yoo, S., Choi, T., Yang, L., and Shin, D · 2015
Earlier work this paper cites.
Diversity networks: Neural network compression using determinantal point processes
Original
Mariet, Z. and Sra, S · 2015
Earlier work this paper cites.
Deep face recognition
Parkhi, O. M., Vedaldi, A., Zisserman, A., et al · 2015
Earlier work this paper cites.
High speed lossless image compression
Siedelmann, H., Wender, A., and Fuchs, M · 2015
Earlier work this paper cites.
Accelerating very deep convolutional networks for classification and detection
Zhang, X., Zou, J., He, K., and Sun, J · 2015
Earlier work this paper cites.
A general simd-based approach to accelerating compression algorithms
Zhao, W. X., Zhang, X., Lemire, D., Shan, D., Nie, J.-Y., Yan, H., and Wen, J.-R · 2015
Earlier work this paper cites.
Perforatedcnns: Acceleration through elimination of redundant convolutions
Figurnov, M., Ibraimova, A., Vetrov, D. P., and Kohli, P · 2016
Earlier work this paper cites.
Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding
Original
Han, S., Mao, H., and Dally, W. J · 2016
Earlier work this paper cites.

Similar

Fast convnets using group-wise brain damage
Lebedev, V. and Lempitsky, V · 2016
Cited alongside, same era.
Pruning filters for efficient convnets
Original
Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P · 2016
Cited alongside, same era.
Pruning convolutional neural networks for resource efficient inference
Original
Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J · 2016
Cited alongside, same era.
Accelerating convolutional neural networks for mobile applications
Wang, P. and Cheng, J · 2016
Cited alongside, same era.
Learning structured sparsity in deep neural networks
Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H · 2016
Cited alongside, same era.
The power of sparsity in convolutional neural networks
Original
Changpinyo, S., Sandler, M., and Zhmoginov, A · 2017
Cited alongside, same era.
More is less: A more complicated network with less inference complexity
Dong, X., Huang, J., Yang, Y., and Yan, S · 2017
Cited alongside, same era.
Gpu kernels for block-sparse weights
Original
Gray, S., Radford, A., and Kingma, D. P · 2017
Cited alongside, same era.
Channel pruning for accelerating very deep neural networks
He, Y., Zhang, X., and Sun, J · 2017
Cited alongside, same era.
Learning efficient convolutional networks through network slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C · 2017
Cited alongside, same era.
Bayesian compression for deep learning
Louizos, C., Ullrich, K., and Welling, M · 2017
Cited alongside, same era.
Thinet: A filter level pruning method for deep neural network compression
Luo, J.-H., Wu, J., and Lin, W · 2017
Cited alongside, same era.

Then

Extreme network compression via filter group approximation
Peng, B., Tan, W., Li, Z., Zhang, S., Xie, D., and Pu, S · 2018
Later among the works it cites.
Network compression using correlation analysis of layer responses
Suau, X., Zappella, L., and Apostoloff, N · 2018
Later among the works it cites.
Spectral-pruning: Compressing deep neural network via spectral analysis
Original
Suzuki, T., Abe, H., Murata, T., Horiuchi, S., Ito, K., Wachi, T., Hirai, S., Yukishima, M., and Nishimura, T · 2018
Later among the works it cites.
Pcas: Pruning channels with attention statistics
Original
Yamamoto, K. and Maeno, K · 2018
Later among the works it cites.
Balanced sparsity for efficient dnn inference on gpu
Original
Yao, Z., Cao, S., and Xiao, W · 2018
Later among the works it cites.
Nisp: Pruning networks using neuron importance score propagation
Yu, R., Li, A., Chen, C.-F., Lai, J.-H., Morariu, V. I., Han, X., Gao, M., Lin, C.-Y., and Davis, L. S · 2018
Later among the works it cites.
https://github.com/BVLC/caffe/issues/4202 , 5 2016
What’s the advantage of the reference caffenet in comparison with the alexnet? · 2019
Later among the works it cites.
https://github.com/keras-team/keras/issues/7848 , 9 2017
Keras exported model shows very low accuracy in tensorflow serving · 2019
Later among the works it cites.
Jointly sparse convolutional neural networks in dual spatial-winograd domains
Original
Choi, Y., El-Khamy, M., and Lee, J · 2019
Later among the works it cites.
Accuracy of resnet50 is much higher than reported!
Crall, J · 2019
Later among the works it cites.
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Frankle, J. and Carbin, M · 2019
Later among the works it cites.
The lottery ticket hypothesis at scale
Original
Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M · 2019
Later among the works it cites.
The state of sparsity in deep neural networks, 2019
Gale, T., Elsen, E., and Hooker, S · 2019
Later among the works it cites.
Training lenet on mnist with caffe
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T · 2019
Later among the works it cites.
Validating resnet50
Jogeshwar, A · 2019
Later among the works it cites.
The mnist database of handwritten digits, 1998b
LeCun, Y., Cortes, C., and Burges, C · 2019
Later among the works it cites.
Snip: single-shot network pruning based on connection sensitivity
Lee, N., Ajanthan, T., and Torr, P. H. S · 2019
Later among the works it cites.
Rethinking the value of network pruning
Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell, T · 2019
Later among the works it cites.
Keras doesn’t reproduce caffe example code accuracy
Nola, D · 2019
Later among the works it cites.
Towards reproducibility: Benchmarking keras and pytorch
Northcutt, C · 2019
Later among the works it cites.
Efficientnet: Rethinking model scaling for convolutional neural networks
Original
Tan, M. and Le, Q. V · 2019
Later among the works it cites.
Change bn layer to use moving mean/var if frozen
Vryniotis, V · 2019
Later among the works it cites.
92.45% on cifar-10 in torch
Zagoruyko, S · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…