Fetching the paper…

Reading the bibliography…

2023

Knowledge Distillation for Efficient Sequences of Training Runs

Liu, Xingyu, Leonardi, Alex, Yu, Lu et al.

Original
PDF
Post on X

Understand

In many practical scenarios -- like hyperparameter search or continual retraining with new data -- related training runs are performed many times in sequence.

Current practice is to train each of these models independently from scratch.
We study the problem of exploiting the computation invested in previous runs to reduce the cost of future runs using knowledge distillation (KD).
We find that augmenting future runs with KD from previous runs dramatically reduces the time necessary to train these models, even taking into account the overhead of KD.

Knowledge Distillation for Efficient Sequences of Training Runs · Around

Built on

Distilling the knowledge in a neural network
Original
Hinton, G., Vinyals, O., Dean, J., et al · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Learning from multiple teacher networks
You, S., Xu, C., Xu, C., and Tao, D · 2017
Earlier work this paper cites.

Similar

Critical learning periods in deep networks
Achille, A., Rovere, M., and Soatto, S · 2019
Cited alongside, same era.
Ensemble knowledge distillation for learning improved and efficient networks
Original
Asif, U., Tang, J., and Harrer, S · 2019
Cited alongside, same era.
A two-teacher framework for knowledge distillation
Chen, X., Su, J., and Zhang, J · 2019
Cited alongside, same era.

Then

Towards understanding ensemble, knowledge distillation and self-distillation in deep learning
Original
Allen-Zhu, Z. and Li, Y · 2020
Later among the works it cites.
On warm-starting neural network training
Ash, J. and Adams, R. P · 2020
Later among the works it cites.
Feature-level ensemble knowledge distillation for aggregating knowledge from multiple networks
Park, S. and Kwak, N · 2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…