2015

Tensorizing Neural Networks

Novikov, Alexander, Podoprikhin, Dmitry, Osokin, Anton et al.

Understand

Deep neural networks currently demonstrate state-of-the-art performance in several domains.

At the same time, models of this class are very demanding in terms of computational resources.
In particular, a large amount of memory is required by commonly used fully-connected layers, making it hard to use the models on low-end devices and stopping the further increase of the model size.
In this paper we convert the dense weight matrices of the fully-connected layers to the Tensor Train format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved.

Built on

L. R. Tucker, “Some mathematical notes on three-mode factor analysis,” Psychometrika , vol. 31, no. 3, pp. 279–311, 1966
1966
Earlier work this paper cites.
J. D. Caroll and J. J. Chang, “Analysis of individual differences in multidimensional scaling via n-way generalization of Eckart-Young decomposition,” Psychometrika , vol. 35, pp. 283–319, 1970
1970
Earlier work this paper cites.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature , vol. 323, no. 6088, pp. 533––536, 1986
1986
Earlier work this paper cites.
G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of control, signals and systems , pp. 303–314, 1989
1989
Earlier work this paper cites.
K. Asanović and N. Morgan, “Experimental determination of precision requirements for back-propagation training of artificial neural networks,” International Computer Science Institute, Tech. Rep., 1991
1991
Earlier work this paper cites.
Y. LeCun, C. Cortes, and C. J. C. Burges, “The MNIST database of handwritten digits,” 1998
1998
Earlier work this paper cites.
W. Hackbusch and S. Kühn, “A new scheme for the tensor representation,” J. Fourier Anal. Appl. , vol. 15, pp. 706–722, 2009
2009
Earlier work this paper cites.
A. Krizhevsky, “Learning multiple layers of features from tiny images,” Master’s thesis, Computer Science Department, University of Toronto, 2009
2009
Earlier work this paper cites.
I. V. Oseledets, “Tensor-Train decomposition,” SIAM J. Scientific Computing , vol. 33, no. 5, pp. 2295–2317, 2011
2011
Earlier work this paper cites.

Similar

E. Gilboa, Y. Saatçi, and J. P. Cunningham, “Scaling multidimensional inference for structured gaussian processes,” arXiv preprint , no. 1209.4120, 2012
2012
Cited alongside, same era.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25 (NIPS) , 2012, pp. 1097–1105
2012
Cited alongside, same era.
J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” in Advances in Neural Information Processing Systems 25 (NIPS) , 2012, pp. 2951–2959
2012
Cited alongside, same era.
M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. de Freitas, “Predicting parameters in deep learning,” in Advances in Neural Information Processing Systems 26 (NIPS) , 2013, pp. 2148–2156
2013
Cited alongside, same era.
I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, “Maxout networks,” in International Conference on Machine Learning (ICML) , 2013, pp. 1319–1327
2013
Cited alongside, same era.
T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, “Low-rank matrix factorization for deep neural network training with high-dimensional output targets,” in International Conference of Acoustics, Speech, and Signal Processing (ICASSP) , 2013, pp. 6655–6659
2013
Cited alongside, same era.
J. Xue, J. Li, and Y. Gong, “Restructuring of deep neural network acoustic models with singular value decomposition,” in Interspeech , 2013, pp. 2365–2369
2013
Cited alongside, same era.
J. Ba and R. Caruana, “Do deep nets really need to be deep?” in Advances in Neural Information Processing Systems 27 (NIPS) , 2014, pp. 2654–2662
2014
Cited alongside, same era.
E. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, “Exploiting linear structure within convolutional networks for efficient evaluation,” in Advances in Neural Information Processing Systems 27 (NIPS) , 2014, pp. 1269–1277
2014
Cited alongside, same era.

Then

Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep convolutional networks using vector quantization,” arXiv preprint , no. 1412.6115, 2014
2014
Later among the works it cites.
V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky, “Speeding-up convolutional neural networks using fine-tuned CP-decomposition,” in International Conference on Learning Representations (ICLR) , 2014
2014
Later among the works it cites.
A. Novikov, A. Rodomanov, A. Osokin, and D. Vetrov, “Putting MRFs on a Tensor Train,” in International Conference on Machine Learning (ICML) , 2014, pp. 811–819
2014
Later among the works it cites.
Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, and Z. Wang, “Deep fried convnets,” arXiv preprint , no. 1412.7149, 2014
2014
Later among the works it cites.
Z. Zhang, X. Yang, I. V. Oseledets, G. E. Karniadakis, and L. Daniel, “Enabling high-dimensional hierarchical uncertainty quantification by ANOVA and tensor-train decomposition,” Computer-Aided Design of Integrated Circuits and Systems, IEEE Transactions on , pp. 63–76, 2014
2014
Later among the works it cites.
W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, “Compressing neural networks with the hashing trick,” in International Conference on Machine Learning (ICML) , 2015, pp. 2285–2294
2015
Closest in time.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” International Journal of Computer Vision (IJCV) , 2015
2015
Closest in time.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations (ICLR) , 2015
2015
Closest in time.
A. Vedaldi and K. Lenc, “Matconvnet – convolutional neural networks for MATLAB,” in Proceeding of the ACM Int. Conf. on Multimedia , 2015
2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…