2019

Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks

You, Haoran, Li, Chaojian, Xu, Pengfei et al.

Understand

(Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations.

However, the identification of these winning tickets still requires the costly train-prune-retrain process, limiting their practical benefits.
In this paper, we discover for the first time that the winning tickets can be identified at the very early training stage, which we term as early-bird (EB) tickets, via low-cost training schemes (e.g., early stopping and low-precision training) at large learning rates.
Our finding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early.

Built on

Energy and policy considerations for deep learning in NLP
Original
Emma Strubell, Ananya Ganesh, and Andrew McCallum · 1906
Earlier work this paper cites.
Dual dynamic inference: Enabling more efficient, adaptive and controllable deep inference
Original
Yue Wang, Jianghao Shen, Ting-Kuei Hu, Pengfei Xu, Tan Nguyen, Richard Baraniuk, Zhangyang Wang, and Yingyan Lin · 1907
Earlier work this paper cites.
Training high-performance and large-scale deep neural networks with full 8-bit integers
Original
Yukuan Yang, Shuang Wu, Lei Deng, Tianyi Yan, Yuan Xie, and Guoqi Li · 1909
Earlier work this paper cites.
Exploiting linear structure within convolutional networks for efficient evaluation
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Deep learning with limited numerical precision
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan · 2015
Earlier work this paper cites.
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Original
Song Han, Huizi Mao, and William J Dally · 2015
Earlier work this paper cites.
Learning structured sparsity in deep neural networks
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li · 2016
Earlier work this paper cites.
Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes
Original
Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda · 2017
Earlier work this paper cites.
Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks
Y. H. Chen, T. Krishna, J. S. Emer, and V. Sze · 2017
Earlier work this paper cites.
Powerai ddl
Original
Minsik Cho, Ulrich Finkler, Sameer Kumar, David S. Kung, Vaibhav Saxena, and Dheeraj Sreedhar · 2017
Earlier work this paper cites.
Accurate, large minibatch SGD: training imagenet in 1 hour
Original
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He · 2017
Earlier work this paper cites.

Similar

Quantized neural networks: Training neural networks with low precision weights and activations
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio · 2017
Cited alongside, same era.
Pruning filters for efficient convnets
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf · 2017
Cited alongside, same era.
Learning efficient convolutional networks through network slimming
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang · 2017
Cited alongside, same era.
Thinet: A filter level pruning method for deep neural network compression
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin · 2017
Cited alongside, same era.
Neural architecture search with reinforcement learning
Original
Barret Zoph and Quoc V. Le · 2017
Cited alongside, same era.
Scalable methods for 8-bit training of neural networks
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry · 2018
Cited alongside, same era.
Soft filter pruning for accelerating deep convolutional neural networks
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang · 2018
Cited alongside, same era.
Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes
Original
Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, et al · 2018
Cited alongside, same era.
Imagenet training in minutes
Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer · 2018
Cited alongside, same era.
Critical learning periods in deep networks
Alessandro Achille, Matteo Rovere, and Stefano Soatto · 2019
Cited alongside, same era.
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Jonathan Frankle and Michael Carbin · 2019
Cited alongside, same era.
Snip: Single-shot network pruning based on connection sensitivity
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr · 2019
Cited alongside, same era.

Then

Towards explaining the regularization effect of initial large learning rate in training neural networks
Original
Yuanzhi Li, Colin Wei, and Tengyu Ma · 2019
Closest in time.
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian · 2019
Closest in time.
NVIDIA Jetson TX2
NVIDIA Inc · 2019
Closest in time.
On the spectral bias of neural networks
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville · 2019
Closest in time.
Low-memory neural network training: A technical report
Original
Nimit Sharad Sohoni, Christopher Richard Aberger, Megan Leszczynski, Jian Zhang, and Christopher Ré · 2019
Closest in time.
Frequency principle: Fourier analysis sheds light on deep neural networks
Original
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma · 2019
Closest in time.
SWALP : Stochastic weight averaging in low precision training
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Christopher De Sa · 2019
Closest in time.
Deconstructing lottery tickets: Zeros, signs, and the supermask
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski · 2019
Closest in time.
The early phase of neural network training
Jonathan Frankle, David J. Schwab, and Ari S. Morcos · 2020
Closest in time.
Comparing rewinding and fine-tuning in neural network pruning
Alex Renda, Jonathan Frankle, and Michael Carbin · 2020
Closest in time.
Fractional Skipping: Towards Finer-Grained Dynamic Inference
Jianghao Shen, Yonggan Fu, Yue Wang, Pengfei Xu, Zhangyang Wang, and Yingyan Lin · 2020
Closest in time.
AutoDNNchip: An automated dnn chip predictor and builderfor both fpgas and asics
Pengfei Xu, Xiaofan Zhang, Hao Cong, Yang Zhao, Yongan Zhang, Chaojian Li, Zetong Guan, Deming Chen, and Yingyan Lin · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…