2017

Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units

Shi, Shaohuai, Chu, Xiaowen

Understand

Rectifier neuron units (ReLUs) have been widely used in deep convolutional networks.

An ReLU converts negative values to zeros, and does not change positive values, which leads to a high sparsity of neurons.
In this work, we first examine the sparsity of the outputs of ReLUs in some popular deep convolutional architectures.
And then we use the sparsity property of ReLUs to accelerate the calculation of convolution by skipping calculations of zero-valued neurons.

Built on

Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition
Kunihiko Fukushima and Sei Miyake · 1982
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.
Characterizing the sparseness of neural codes
Benjamin Willmore and David J Tolhurst · 2001
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton · 2010
Earlier work this paper cites.
Deep sparse rectifier neural networks
Xavier Glorot, Antoine Bordes, and Yoshua Bengio · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Rectifier nonlinearities improve neural network acoustic models
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng · 2013
Earlier work this paper cites.
Dadiannao: A machine-learning supercomputer
Yunji Chen, Tao Luo, Shaoli Liu, Shijin Zhang, Liqiang He, Jia Wang, Ling Li, Tianshi Chen, Zhiwei Xu, Ninghui Sun, et al · 2014
Earlier work this paper cites.
cudnn: Efficient primitives for deep learning
Original
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer · 2014
Earlier work this paper cites.

Similar

Caffe: Convolutional architecture for fast feature embedding
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell · 2014
Cited alongside, same era.
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Original
Song Han, Huizi Mao, and William J Dally · 2015
Cited alongside, same era.
Learning both weights and connections for efficient neural network
Song Han, Jeff Pool, John Tran, and William Dally · 2015
Cited alongside, same era.
Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton · 2015
Cited alongside, same era.
Sparse convolutional neural networks
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky · 2015
Cited alongside, same era.
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich · 2015
Cited alongside, same era.
Cnvlutin: Ineffectual-neuron-free deep neural network computing
Jorge Albericio, Patrick Judd, Tayler Hetherington, Tor Aamodt, Natalie Enright Jerger, and Andreas Moshovos · 2016
Cited alongside, same era.
Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks
Yu-Hsin Chen, Tushar Krishna, Joel S Emer, and Vivienne Sze · 2016
Cited alongside, same era.
Dynamic network surgery for efficient dnns
Yiwen Guo, Anbang Yao, and Yurong Chen · 2016
Cited alongside, same era.

Then

Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Later among the works it cites.
Fast algorithms for convolutional neural networks
Andrew Lavin and Scott Gray · 2016
Later among the works it cites.
Minerva: Enabling low-power, highly-accurate deep neural network accelerators
Brandon Reagen, Paul Whatmough, Robert Adolf, Saketh Rama, Hyunkwang Lee, Sae Kyu Lee, José Miguel Hernández-Lobato, Gu-Yeon Wei, and David Brooks · 2016
Later among the works it cites.
Benchmarking state-of-the-art deep learning software tools
Original
Shaohuai Shi, Qiang Wang, Pengfei Xu, and Xiaowen Chu · 2016
Later among the works it cites.
Learning structured sparsity in deep neural networks
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li · 2016
Later among the works it cites.
The power of sparsity in convolutional neural networks
Original
Soravit Changpinyo, Mark Sandler, and Andrey Zhmoginov · 2017
Closest in time.
Faster cnns with direct sparse convolutions and guided pruning
Wei Wen Ping Tak Peter Tang Hai Li Yiran Chen Jongsoo Park, Sheng Li · 2017
Closest in time.
Efficient processing of deep neural networks: A tutorial and survey
Original
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel Emer · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…