2018

Snapshot Distillation: Teacher-Student Optimization in One Generation

Yang, Chenglin, Xie, Lingxi, Su, Chi et al.

Understand

Optimizing a deep neural network is a fundamental task in computer vision, yet direct training methods often suffer from over-fitting.

Teacher-student optimization aims at providing complementary cues from a model trained previously, but these approaches are often considerably slow due to the pipeline of training a few generations in sequence, i.e., time complexity is increased by several times.
This paper presents snapshot distillation (SD), the first framework which enables teacher-student optimization in one generation.
The idea of SD is very simple: instead of borrowing supervision signals from previous generations, we extract such information from earlier epochs in the same generation, meanwhile make sure that the difference between teacher and student is sufficiently large so as to prevent under-fitting.

Built on

Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky and G. Hinton · 2009
Earlier work this paper cites.
What does classifying more than 10,000 image categories tell us?
J. Deng, A. C. Berg, K. Li, and L. Fei-Fei · 2010
Earlier work this paper cites.
The pascal visual object classes (voc) challenge
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman · 2010
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
V. Nair and G. E. Hinton · 2010
Earlier work this paper cites.
Improving the fisher kernel for large-scale image classification
F. Perronnin, J. Sanchez, and T. Mensink · 2010
Earlier work this paper cites.
Semantic contours from inverse detectors
B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, and J. Malik · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Learning hierarchical similarity metrics
N. Verma, D. Mahajan, S. Sellamanickam, and V. Nair · 2012
Earlier work this paper cites.
Decaf: A deep convolutional activation feature for generic visual recognition
J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell · 2014
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
R. Girshick, J. Donahue, T. Darrell, and J. Malik · 2014
Earlier work this paper cites.
Cnn features off-the-shelf: an astounding baseline for recognition
A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson · 2014
Earlier work this paper cites.
Fitnets: Hints for thin deep nets
A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio · 2014
Earlier work this paper cites.
Dropout: A simple way to prevent neural networks from overfitting
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov · 2014
Earlier work this paper cites.
Learning fine-grained image similarity with deep ranking
J. Wang, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, Y. Wu, et al · 2014
Earlier work this paper cites.
Fast r-cnn
R. Girshick · 2015
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
G. Hinton, O. Vinyals, and J. Dean · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
S. Ioffe and C. Szegedy · 2015
Earlier work this paper cites.
Deep learning
Y. LeCun, Y. Bengio, and G. E. Hinton · 2015
Earlier work this paper cites.

Similar

Fully convolutional networks for semantic segmentation
J. Long, E. Shelhamer, and T. Darrell · 2015
Cited alongside, same era.
Faster r-cnn: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al · 2015
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
K. Simonyan and A. Zisserman · 2015
Cited alongside, same era.
Going deeper with convolutions
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, et al · 2015
Cited alongside, same era.
Holistically-nested edge detection
S. Xie and Z. Tu · 2015
Cited alongside, same era.
Label-embedding for image classification
Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid · 2016
Cited alongside, same era.
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs
L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille · 2016
Cited alongside, same era.
Net2net: Accelerating learning via knowledge transfer
T. Chen, I. Goodfellow, and J. Shlens · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Sgdr: Stochastic gradient descent with warm restarts
Original
I. Loshchilov and F. Hutter · 2016
Cited alongside, same era.
Rethinking the inception architecture for computer vision
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna · 2016
Cited alongside, same era.

Then

Regularizing neural networks by penalizing confident output distributions
Original
G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton · 2017
Later among the works it cites.
Super-convergence: Very fast training of residual networks using large learning rates
Original
L. N. Smith and N. Topin · 2017
Later among the works it cites.
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
A. Tarvainen and H. Valpola · 2017
Later among the works it cites.
Hierarchical loss for classification
Original
C. Wu, M. Tygert, and Y. LeCun · 2017
Later among the works it cites.
Genetic cnn
L. Xie and A. Yuille · 2017
Later among the works it cites.
Aggregated residual transformations for deep neural networks
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He · 2017
Later among the works it cites.
A gift from knowledge distillation: Fast optimization, network minimization and transfer learning
J. Yim, D. Joo, J. Bae, and J. Kim · 2017
Later among the works it cites.
mixup: Beyond empirical risk minimization
Original
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz · 2017
Later among the works it cites.
Interleaved group convolutions
T. Zhang, G. J. Qi, B. Xiao, and J. Wang · 2017
Later among the works it cites.
Deep mutual learning
Original
Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu · 2017
Later among the works it cites.
Random erasing data augmentation
Original
Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang · 2017
Later among the works it cites.
Neural architecture search with reinforcement learning
B. Zoph and Q. V. Le · 2017
Later among the works it cites.
Label refinery: Improving imagenet classification through label progression
Original
H. Bagherinezhad, M. Horton, M. Rastegari, and A. Farhadi · 2018
Closest in time.
Born again neural networks
T. Furlanello, Z. C. Lipton, L. Itti, and A. Anandkumar · 2018
Closest in time.
Squeeze-and-excitation networks
J. Hu, L. Shen, and G. Sun · 2018
Closest in time.
Snapshot ensembles: Train 1, get m for free
G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger · 2018
Closest in time.
Progressive neural architecture search
C. Liu, B. Zoph, J. Shlens, W. Hua, L. J. Li, L. Fei-Fei, A. L. Yuille, J. Huang, and K. Murphy · 2018
Closest in time.
Knowledge distillation in generations: More tolerant teachers educate better students
Original
C. Yang, L. Xie, S. Qiao, and A. L. Yuille · 2018
Closest in time.
Image-level classification by hierarchical structure learning with visual and semantic similarities
C. Zhang, J. Cheng, and Q. Tian · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…