2016

Pruning Filters for Efficient ConvNets

Li, Hao, Kadav, Asim, Durdanovic, Igor et al.

Understand

The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs.

Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy.
However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks.
We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy.

Built on

Optimal Brain Damage
Yann Le Cun, John S Denker, and Sara A Solla · 1989
Earlier work this paper cites.
Second Order Derivatives for Network Pruning: Optimal Brain Surgeon
Babak Hassibi and David G Stork · 1993
Earlier work this paper cites.
Torch7: A matlab-like environment for machine learning
Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet · 2011
Earlier work this paper cites.
Imagenet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Predicting parameters in deep learning
Misha Denil, Babak Shakibi, Laurent Dinh, Nando de Freitas, et al · 2013
Earlier work this paper cites.
Network in Network
Original
Min Lin, Qiang Chen, and Shuicheng Yan · 2013
Earlier work this paper cites.
Fast Training of Convolutional Networks through FFTs
Original
Michael Mathieu, Mikael Henaff, and Yann LeCun · 2013
Earlier work this paper cites.
Speeding up convolutional neural networks with low rank expansions
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman · 2014
Earlier work this paper cites.
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Earlier work this paper cites.
Visualizing and Understanding Convolutional Networks
Matthew D Zeiler and Rob Fergus · 2014
Earlier work this paper cites.
Structured Pruning of Deep Convolutional Neural Networks
Original
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung · 2015
Earlier work this paper cites.

Similar

Learning both Weights and Connections for Efficient Neural Network
Song Han, Jeff Pool, John Tran, and William Dally · 2015
Cited alongside, same era.
Convolutional Neural Networks at Constrained Time Cost
Kaiming He and Jian Sun · 2015
Cited alongside, same era.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe and Christian Szegedy · 2015
Cited alongside, same era.
Sparse Convolutional Neural Networks
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky · 2015
Cited alongside, same era.
Channel-Level Acceleration of Deep Face Representations
Adam Polyak and Lior Wolf · 2015
Cited alongside, same era.
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei · 2015
Cited alongside, same era.
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan and Andrew Zisserman · 2015
Cited alongside, same era.
Data-free Parameter Pruning for Deep Neural Networks
Suraj Srinivas and R Venkatesh Babu · 2015
Cited alongside, same era.
92.45% on CIFAR-10 in Torch
Sergey Zagoruyko · 2015
Cited alongside, same era.
Binarynet: Training deep neural networks with weights and activations constrained to+ 1 or-1
Original
Matthieu Courbariaux and Yoshua Bengio · 2016
Cited alongside, same era.
EIE: Efficient Inference Engine on Compressed Deep Neural Network
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally
Cited in the paper.
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
Song Han, Huizi Mao, and William J Dally
Cited in the paper.

Then

Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Closest in time.
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and ¡ 1MB model size
Original
Forrest Iandola, Matthew Moskewicz, Khalidand Ashraf, Song Han, William Dally, and Keutzer Kurt · 2016
Closest in time.
Training CNNs with Low-Rank Filters for Efficient Image Classification
Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi · 2016
Closest in time.
Fast Algorithms for Convolutional Neural Networks
Andrew Lavin and Scott Gray · 2016
Closest in time.
Fast Convnets Using Group-wise Brain Damage
Vadim Lebedev and Victor Lempitsky · 2016
Closest in time.
Diversity Networks
Zelda Mariet and Suvrit Sra · 2016
Closest in time.
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi · 2016
Closest in time.
Convolutional neural networks with low-rank regularization
Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E · 2016
Closest in time.
Learning Structured Sparsity in Deep Learning
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li · 2016
Closest in time.
Less Is More: Towards Compact CNNs
Hao Zhou, Jose Alvarez, and Fatih Porikli · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…