2019

Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network

Dong, Bin, Hou, Jikai, Lu, Yiping et al.

Understand

Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity.

In this paper, we aim to provide a theoretical understanding on what mainly helps with the distillation.
Our answer is "early stopping".
Assuming that the teacher network is overparameterized, we argue that the teacher network is essentially harvesting dark knowledge from the data via early stopping.

Built on

Iterative methods by space decomposition and subspace correction
Jinchao Xu · 1992
Earlier work this paper cites.
Inverse scale space theory for inverse problems
Otmar Scherzer and Chuck Groetsch · 2001
Earlier work this paper cites.
Nonlinear inverse scale space methods for image restoration
Martin Burger, Stanley Osher, Jinjun Xu, and Guy Gilboa · 2005
Earlier work this paper cites.
Learning from examples as an inverse problem
Ernesto De Vito, Lorenzo Rosasco, Andrea Caponnetto, Umberto De Giovannini, and Francesca Odone · 2005
Earlier work this paper cites.
Nonlinear inverse scale space methods
Martin Burger, Guy Gilboa, Stanley Osher, Jinjun Xu, et al · 2006
Earlier work this paper cites.
Learning theory estimates via integral operators and their approximations
Steve Smale and Ding-Xuan Zhou · 2007
Earlier work this paper cites.
Iterative regularization and nonlinear inverse scale space applied to wavelet-based denoising
Jinjun Xu and Stanley Osher · 2007
Earlier work this paper cites.
On early stopping in gradient descent learning
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto · 2007
Earlier work this paper cites.
A nonlinear inverse scale space method for a convex multiplicative noise model
Jianing Shi and Stanley Osher · 2008
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Earlier work this paper cites.
Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton · 2015
Earlier work this paper cites.
Deeply-supervised nets
Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu · 2015
Earlier work this paper cites.
Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer
Original
Sergey Zagoruyko and Nikos Komodakis · 2016
Earlier work this paper cites.
Wide residual networks
Original
Sergey Zagoruyko and Nikos Komodakis · 2016
Earlier work this paper cites.
Understanding deep learning requires rethinking generalization
Original
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals · 2016
Earlier work this paper cites.
A closer look at memorization in deep networks
Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al · 2017
Earlier work this paper cites.
Sobolev training for neural networks
Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan Pascanu · 2017
Earlier work this paper cites.

Similar

Shake-shake regularization
Original
Xavier Gastaldi · 2017
Cited alongside, same era.
Like what you like: Knowledge distill via neuron selectivity transfer
Original
Zehao Huang and Naiyan Wang · 2017
Cited alongside, same era.
Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels
Original
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei · 2017
Cited alongside, same era.
Learning from noisy singly-labeled data
Original
Ashish Khetan, Zachary C Lipton, and Anima Anandkumar · 2017
Cited alongside, same era.
Sgdr: Stochastic gradient descent with warm restarts
Ilya Loshchilov and Frank Hutter · 2017
Cited alongside, same era.
Deep learning is robust to massive label noise
Original
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit · 2017
Cited alongside, same era.
A convergence theory for deep learning via over-parameterization
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song · 2018
Cited alongside, same era.
On the optimization of deep networks: Implicit acceleration by overparameterization
Original
Sanjeev Arora, Nadav Cohen, and Elad Hazan · 2018
Cited alongside, same era.
Label refinery: Improving imagenet classification through label progression
Original
Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, and Ali Farhadi · 2018
Cited alongside, same era.
Saas: Speed as a supervisor for semi-supervised learning
Safa Cicek, Alhussein Fawzi, and Stefano Soatto · 2018
Cited alongside, same era.
Gradient descent finds global minima of deep neural networks
Original
Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai · 2018
Cited alongside, same era.
Gradient descent provably optimizes over-parameterized neural networks
Original
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh · 2018
Cited alongside, same era.

Then

Fluid directed rigid body control using deep reinforcement learning
Pingchuan Ma, Yunsheng Tian, Zherong Pan, Bo Ren, and Dinesh Manocha · 2018
Later among the works it cites.
Dimensionality-driven learning with noisy labels
Original
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey · 2018
Later among the works it cites.
Foundations of machine learning
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar · 2018
Later among the works it cites.
Towards understanding the role of over-parametrization in generalization of neural networks
Original
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro · 2018
Later among the works it cites.
Overparameterized nonlinear learning: Gradient descent takes the shortest path?
Original
Samet Oymak and Mahdi Soltanolkotabi · 2018
Later among the works it cites.
On the spectral bias of deep neural networks
Original
Nasim Rahaman, Devansh Arpit, Aristide Baratin, Felix Draxler, Min Lin, Fred A Hamprecht, Yoshua Bengio, and Aaron Courville · 2018
Later among the works it cites.
Joint optimization framework for learning with noisy labels
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa · 2018
Later among the works it cites.
Knowledge distillation in generations: More tolerant teachers educate better students
Original
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan Yuille · 2018
Later among the works it cites.
Snapshot distillation: Teacher-student optimization in one generation
Original
Chenglin Yang, Lingxi Xie, Chi Su, and Alan L Yuille · 2018
Later among the works it cites.
Generalized cross entropy loss for training deep neural networks with noisy labels
Zhilu Zhang and Mert Sabuncu · 2018
Later among the works it cites.
Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks
Original
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang · 2019
Closest in time.
Pushing the right boundaries matters! wasserstein adversarial training for label noise
Original
Bharath Bhushan Damodaran, Kilian Fatras, Sylvain Lobry, Rémi Flamary, Devis Tuia, and Nicolas Courty · 2019
Closest in time.
Understanding generalization of deep neural networks trained with noisy labels
Original
Wei Hu, Zhiyuan Li, and Dingli Yu · 2019
Closest in time.
Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks
Original
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak · 2019
Closest in time.
A priori estimates of the population risk for residual networks
Original
Chao Ma, Qingcan Wang, et al · 2019
Closest in time.
Frequency principle: Fourier analysis sheds light on deep neural networks
Original
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma · 2019
Closest in time.
How does disagreement benefit co-teaching?
Original
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…