2019

Jointly Sparse Convolutional Neural Networks in Dual Spatial-Winograd Domains

Choi, Yoojin, El-Khamy, Mostafa, Lee, Jungwon

Understand

We consider the optimization of deep convolutional neural networks (CNNs) such that they provide good performance while having reduced complexity if deployed on either conventional systems with spatial-domain convolution or lower-complexity systems designed for Winograd convolution.

The proposed framework produces one compressed model whose convolutional filters can be made sparse either in the spatial domain or in the Winograd domain.
Hence, the compressed model can be deployed universally on any platform, without need for re-training on the deployed platform.
To get a better compression ratio, the sparse model is compressed in the spatial domain that has a fewer number of parameters.

Built on

“A universal algorithm for sequential data compression,”
Jacob Ziv and Abraham Lempel, · 1977
Earlier work this paper cites.
“Compression of individual sequences via variable-rate coding,”
Jacob Ziv and Abraham Lempel, · 1978
Earlier work this paper cites.
Arithmetic Complexity of Computations
Shmuel Winograd, · 1980
Earlier work this paper cites.
“A technique for high-performance data compression,”
Terry A. Welch, · 1984
Earlier work this paper cites.
“On universal quantization,”
Jacob Ziv, · 1985
Earlier work this paper cites.
“Uncertainty principles and signal recovery,”
David L Donoho and Philip B Stark, · 1989
Earlier work this paper cites.
“Atomic decomposition by basis pursuit,”
Scott Shaobing Chen, David L Donoho, and Michael A Saunders, · 2001
Earlier work this paper cites.
“Universal lossless source coding with the Burrows Wheeler transform,”
Michelle Effros, Karthik Visweswariah, Sanjeev R Kulkarni, and Sergio Verdú, · 2002
Earlier work this paper cites.
Fast Algorithms for Signal Processing
Richard E Blahut, · 2010
Earlier work this paper cites.
“Fast training of convolutional networks through FFTs,”
Original
Michael Mathieu, Mikael Henaff, and Yann LeCun, · 2013
Earlier work this paper cites.
“Fast convolutional nets with fbfft: A GPU performance evaluation,”
Original
Nicolas Vasilache, Jeff Johnson, Michael Mathieu, Soumith Chintala, Serkan Piantino, and Yann LeCun, · 2014
Earlier work this paper cites.

Similar

“Adam: A method for stochastic optimization,”
Original
Diederik Kingma and Jimmy Ba, · 2014
Cited alongside, same era.
“Deep learning,”
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, · 2015
Cited alongside, same era.
“ImageNet large scale visual recognition challenge,”
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al., · 2015
Cited alongside, same era.
“Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding,”
Song Han, Huizi Mao, and William J Dally, · 2016
Cited alongside, same era.
“Fast algorithms for convolutional neural networks,”
Andrew Lavin and Scott Gray, · 2016
Cited alongside, same era.
“Fast convnets using group-wise brain damage,”
Vadim Lebedev and Victor Lempitsky, · 2016
Cited alongside, same era.
“Dynamic network surgery for efficient DNNs,”
Yiwen Guo, Anbang Yao, and Yurong Chen, · 2016
Cited alongside, same era.
“Deep residual learning for image recognition,”
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, · 2016
Cited alongside, same era.
“Efficient processing of deep neural networks: A tutorial and survey,”
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer, · 2017
Cited alongside, same era.
“Towards the limit of network quantization,”
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee, · 2017
Cited alongside, same era.
“Soft weight-sharing for neural network compression,”
Karen Ullrich, Edward Meeds, and Max Welling, · 2017
Cited alongside, same era.

Then

“Soft-to-hard vector quantization for end-to-end learning compressible representations,”
Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc V Gool, · 2017
Later among the works it cites.
“Variational dropout sparsifies deep neural networks,”
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov, · 2017
Later among the works it cites.
“Bayesian compression for deep learning,”
Christos Louizos, Karen Ullrich, and Max Welling, · 2017
Later among the works it cites.
“Runtime neural pruning,”
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou, · 2017
Later among the works it cites.
“Faster CNNs with direct sparse convolutions and guided pruning,”
Jongsoo Park, Sheng Li, Wei Wen, Ping Tak Peter Tang, Hai Li, Yiran Chen, and Pradeep Dubey, · 2017
Later among the works it cites.
“Enabling sparse Winograd convolution by native pruning,”
Original
Sheng Li, Jongsoo Park, and Ping Tak Peter Tang, · 2017
Later among the works it cites.
“Universal deep neural network compression,”
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee, · 2018
Later among the works it cites.
“Compressing neural networks using the variational information bottleneck,”
Bin Dai, Chen Zhu, and David Wipf, · 2018
Later among the works it cites.
“Efficient sparse-Winograd convolutional neural networks,”
Xingyu Liu, Jeff Pool, Song Han, and William J Dally, · 2018
Later among the works it cites.
“Learning structured sparsity in deep neural networks,”
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li, · 2082
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…