2016

Ultimate tensorization: compressing convolutional and FC layers alike

Garipov, Timur, Podoprikhin, Dmitry, Novikov, Alexander et al.

Understand

Convolutional neural networks excel in image recognition tasks, but this comes at the cost of high computational and memory complexity.

To tackle this problem, [1] developed a tensor factorization framework to compress fully-connected layers.
In this paper, we focus on compressing convolutional layers.
We show that while the direct application of the tensor framework [1] to the 4-dimensional kernel of convolution does compress the layer, we can do better.

Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.
Tensor-Train decomposition
I. V. Oseledets · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Restructuring of deep neural network acoustic models with singular value decomposition
J. Xue, J. Li, and Y. Gong · 2013
Earlier work this paper cites.
Low-rank matrix factorization for deep neural network training with high-dimensional output targets
T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran · 2013
Earlier work this paper cites.
Predicting parameters in deep learning
M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. de Freitas · 2013
Earlier work this paper cites.
A convolutional neural network for modelling sentences
Original
N. Kalchbrenner, E. Grefenstette, and P. Blunsom · 2014
Earlier work this paper cites.
Caffe: Convolutional architecture for fast feature embedding
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, et al · 2014
Earlier work this paper cites.

Tensorizing neural networks
A. Novikov, D. Podoprikhin, A. Osokin, and D. Vetrov · 2015
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
K. Simonyan and A. Zisserman · 2015
Cited alongside, same era.
Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding
Original
S. Han, H. Mao, and W. J. Dally · 2015
Cited alongside, same era.
Matconvnet: Convolutional neural networks for matlab
A. Vedaldi and K. Lenc · 2015
Cited alongside, same era.
TensorFlow: Large-scale machine learning on heterogeneous systems, 2015
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, et al · 2015
Cited alongside, same era.
Compressing neural networks with the hashing trick
W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen · 2015
Cited alongside, same era.
Going deeper with convolutions
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, et al · 2015
Cited alongside, same era.

Deep residual learning for image recognition
Original
K. He, X. Zhang, S. Ren, and J. Sun · 2015
Later among the works it cites.
Learning both weights and connections for efficient neural network
S. Han, J. Pool, J. Tran, and W. Dally · 2015
Later among the works it cites.
Speeding-up convolutional neural networks using fine-tuned CP-decomposition
V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky · 2015
Later among the works it cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
S. Ioffe and C. Szegedy · 2015
Later among the works it cites.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, et al · 2015
Later among the works it cites.
PerforatedCNNs: Acceleration through elimination of redundant convolutions
M. Figurnov, A. Ibraimova, D. Vetrov, and P. Kohli · 2016
Closest in time.
Factorized convolutional neural networks
Original
M. Wang, B. Liu, and H. Foroosh · 2016
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…