2018

Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing

Rosenfeld, Amir, Tsotsos, John K.

Understand

Training deep neural networks results in strong learned representations that show good generalization capabilities.

In most cases, training involves iterative modification of all weights inside the network via back-propagation.
In Extreme Learning Machines, it has been suggested to set the first layer of a network to fixed random values instead of learning it.
In this paper, we propose to take this approach a step further and fix almost all layers of a deep convolutional neural network, allowing only a small portion of the weights to be learned.

Built on

P. L. Bartlett, “For valid generalization the size of the weights is more important than the size of the network,” in Advances in neural information processing systems , 1997, pp. 134–140
1997
Earlier work this paper cites.
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” 2009
2009
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems , 2012, pp. 1097–1105
2012
Earlier work this paper cites.
K. J. Horadam, Hadamard matrices and their applications . Princeton university press, 2012
2012
Earlier work this paper cites.
K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” arXiv preprint arXiv:1312.6034 , 2013
Original
2013
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
Original
2014
Earlier work this paper cites.
M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in European conference on computer vision . Springer, 2014, pp. 818–833
2014
Earlier work this paper cites.
D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014
Original
2014
Earlier work this paper cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , “Imagenet large scale visual recognition challenge,” International Journal of Computer Vision , vol. 115, no. 3, pp. 211–252, 2015
2015
Earlier work this paper cites.
G. Huang, G.-B. Huang, S. Song, and K. You, “Trends in extreme learning machines: A review,” Neural Networks , vol. 61, pp. 32–48, 2015
2015
Earlier work this paper cites.
R. Giryes, G. Sapiro, and A. M. Bronstein, “Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?” arXiv preprint arXiv:1504.08291 , 2015
Original
2015
Earlier work this paper cites.
S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149 , 2015
Original
2015
Earlier work this paper cites.
S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in Advances in Neural Information Processing Systems , 2015, pp. 1135–1143
2015
Earlier work this paper cites.
B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Pensky, “Sparse convolutional neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 806–814
2015
Earlier work this paper cites.

Similar

A. Mahendran and A. Vedaldi, “Understanding deep image representations by inverting them,” in Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on . IEEE, 2015, pp. 5188–5196
2015
Cited alongside, same era.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International Conference on Machine Learning , 2015, pp. 448–456
2015
Cited alongside, same era.
M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein, “On the expressive power of deep neural networks,” arXiv preprint arXiv:1606.05336 , 2016
Original
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in European Conference on Computer Vision . Springer, 2016, pp. 630–645
2016
Cited alongside, same era.
S. Zagoruyko and N. Komodakis, “Wide Residual Networks.” CoRR , vol. abs/1605.07146, 2016
Original
2016
Cited alongside, same era.
H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” arXiv preprint arXiv:1608.08710 , 2016
Original
2016
Cited alongside, same era.
F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and¡ 0.5 MB model size,” arXiv preprint arXiv:1602.07360 , 2016
Original
2016
Cited alongside, same era.
M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “Xnor-net: Imagenet classification using binary convolutional neural networks,” in European Conference on Computer Vision . Springer, 2016, pp. 525–542
2016
Cited alongside, same era.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on . IEEE, 2016, pp. 2921–2929
2016
Cited alongside, same era.
J. Zhang, Z. Lin, J. Brandt, X. Shen, and S. Sclaroff, “Top-down neural attention by excitation backprop,” in European Conference on Computer Vision . Springer, 2016, pp. 543–559
2016
Cited alongside, same era.
G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten, “Densely connected convolutional networks,” arXiv preprint arXiv:1608.06993 , 2016
Original
2016
Cited alongside, same era.
R. Shwartz-Ziv and N. Tishby, “Opening the Black Box of Deep Neural Networks via Information,” arXiv preprint arXiv:1703.00810 , 2017
Original
2017
Cited alongside, same era.

Then

A. Rudi and L. Rosasco, “Generalization properties of learning with random features,” in Advances in Neural Information Processing Systems , 2017, pp. 3218–3228
2017
Later among the works it cites.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861 , 2017
Original
2017
Later among the works it cites.
O. Shayar, D. Levi, and E. Fetaya, “Learning Discrete Weights Using the Local Reparameterization Trick,” arXiv preprint arXiv:1710.07739 , 2017
Original
2017
Later among the works it cites.
M. Biparva and J. Tsotsos, “STNet: Selective Tuning of Convolutional Networks for Object Localization,” in The IEEE International Conference on Computer Vision (ICCV) , vol. 2, 2017
2017
Later among the works it cites.
D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba, “Network Dissection: Quantifying Interpretability of Deep Visual Representations,” arXiv preprint arXiv:1704.05796 , 2017
Original
2017
Later among the works it cites.
Y. Dong, H. Su, J. Zhu, and F. Bao, “Towards interpretable deep neural networks by leveraging adversarial examples,” arXiv preprint arXiv:1708.05493 , 2017
Original
2017
Later among the works it cites.
S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Learning multiple visual domains with residual adapters,” arXiv preprint arXiv:1705.08045 , 2017
Original
2017
Later among the works it cites.
J. Zhang, I. Mitliagkas, and C. Ré, “YellowFin and the Art of Momentum Tuning,” arXiv preprint arXiv:1706.03471 , 2017
Original
2017
Later among the works it cites.
P. Zhou and J. Feng, “The Landscape of Deep Learning Algorithms,” arXiv preprint arXiv:1705.07038 , 2017
Original
2017
Later among the works it cites.
D. Mittal, S. Bhardwaj, M. M. Khapra, and B. Ravindran, “Recovering from Random Pruning: On the Plasticity of Deep Convolutional Neural Networks,” 2018
2018
Closest in time.
E. Hoffer, I. Hubara, and D. Soudry, “Fix your classifier: the marginal value of training the last weight layer,” arXiv preprint arXiv:1801.04540 , 2018
Original
2018
Closest in time.
R. Fong and A. Vedaldi, “Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks,” arXiv preprint arXiv:1801.03454 , 2018
Original
2018
Closest in time.
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured sparsity in deep neural networks,” in Advances in Neural Information Processing Systems , 2016, pp. 2074–2082
2082
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…