2019

What's Hidden in a Randomly Weighted Neural Network?

Ramanujan, Vivek, Wortsman, Mitchell, Kembhavi, Aniruddha et al.

Understand

Training a neural network is synonymous with learning the values of the weights.

By contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values.
Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet.
Not only do these "untrained subnetworks" exist, but we provide an algorithm to effectively find them.

Built on

Learning representations by back-propagating errors
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams · 1986
Earlier work this paper cites.
A simple weight decay can improve generalization
Anders Krogh and John A. Hertz · 1991
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
MNIST handwritten digit database
Yann LeCun and Corinna Cortes · 2010
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Estimating or propagating gradients through stochastic neurons for conditional computation, 2013
Yoshua Bengio, Nicholas Léonard, and Aaron Courville · 2013
Earlier work this paper cites.
Extreme learning machines [trends & controversies]
Erik Cambria, Guang-Bin Huang, Liyanaarachchi Lekamalage Chamara Kasun, Hongming Zhou, Chi Man Vong, Jiarun Lin, Jianping Yin, Zhiping Cai, Qiang Liu, Kuan Li, et al · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization, 2014
Diederik P. Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.

Similar

Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman · 2015
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Sgdr: Stochastic gradient descent with warm restarts, 2016
Ilya Loshchilov and Frank Hutter · 2016
Cited alongside, same era.
Unsupervised learning of visual representations by solving jigsaw puzzles, 2016
Mehdi Noroozi and Paolo Favaro · 2016
Cited alongside, same era.
Wide residual networks, 2016
Sergey Zagoruyko and Nikos Komodakis · 2016
Cited alongside, same era.
Neural architecture search with reinforcement learning
Original
Barret Zoph and Quoc V Le · 2016
Cited alongside, same era.
Mobilenets: Efficient convolutional neural networks for mobile vision applications
Original
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam · 2017
Cited alongside, same era.
Automatic differentiation in pytorch
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer · 2017
Cited alongside, same era.
Snip: Single-shot network pruning based on connection sensitivity
Original
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr · 2018
Cited alongside, same era.
Progressive neural architecture search
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy · 2018
Cited alongside, same era.
Mnasnet: Platform-aware neural architecture search for mobile
Original
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le · 2018
Cited alongside, same era.

Then

ProxylessNAS: Direct neural architecture search on target task and hardware
Han Cai, Ligeng Zhu, and Song Han · 2019
Closest in time.
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Jonathan Frankle and Michael Carbin · 2019
Closest in time.
Weight agnostic neural networks
Adam Gaier and David Ha · 2019
Closest in time.
A signal propagation perspective for pruning neural networks at initialization
Original
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr · 2019
Closest in time.
Deeplearningexamples/pytorch/classification
NVIDIA · 2019
Closest in time.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2019
Closest in time.
Intriguing properties of randomly weighted networks: Generalizing while learning next to nothing
Amir Rosenfeld and John K Tsotsos · 2019
Closest in time.
Efficientnet: Rethinking model scaling for convolutional neural networks
Mingxing Tan and Quoc V. Le · 2019
Closest in time.
Discovering neural wirings
Original
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari · 2019
Closest in time.
Exploring randomly wired neural networks for image recognition
Original
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He · 2019
Closest in time.
Deconstructing lottery tickets: Zeros, signs, and the supermask, 2019
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…