2019

Preparing Lessons: Improve Knowledge Distillation with Better Supervision

Wen, Tiancheng, Lai, Shenqi, Qian, Xueming

Understand

Knowledge distillation (KD) is widely used for training a compact model with the supervision of another large model, which could effectively improve the performance.

Previous methods mainly focus on two aspects: 1) training the student to mimic representation space of the teacher; 2) training the model progressively or adding extra module like discriminator.
Knowledge from teacher is useful, but it is still not exactly right compared with ground truth.
Besides, overly uncertain supervision also influences the result.

Built on

B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi, “A comprehensive overhaul of feature distillation,” in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 1921–1930
1930
Earlier work this paper cites.
C. Bucilua, R. Caruana, and A. Niculescu-Mizil, “Model compression,” in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining , 2006, pp. 535–541
2006
Earlier work this paper cites.
J. Ba and R. Caruana, “Do deep nets really need to be deep?” in Advances in neural information processing systems , 2014, pp. 2654–2662
2014
Earlier work this paper cites.
G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” NIPS Workshop , 2014
2014
Earlier work this paper cites.
A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,” International Conference on Learning Representations , 2015
2015
Earlier work this paper cites.
Y. Le and X. Yang, “Tiny imagenet visual recognition challenge,” CS 231N , 2015
2015
Earlier work this paper cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , “Imagenet large scale visual recognition challenge,” International journal of computer vision , vol. 115, no. 3, pp. 211–252, 2015
2015
Earlier work this paper cites.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 2818–2826
2016
Earlier work this paper cites.
L. Xie, J. Wang, Z. Wei, M. Wang, and Q. Tian, “Disturblabel: Regularizing cnn on the loss layer,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 4753–4762
2016
Earlier work this paper cites.
A. Shrivastava, A. Gupta, and R. Girshick, “Training region-based object detectors with online hard example mining,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 761–769
2016
Earlier work this paper cites.
B. B. Sau and V. N. Balasubramanian, “Deep model compression: Distilling knowledge from noisy teachers,” arXiv preprint arXiv:1610.09650 , 2016
Original
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in European conference on computer vision . Springer, 2016, pp. 630–645
2016
Earlier work this paper cites.
——, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” arXiv preprint arXiv:1608.03983 , 2016
Original
2016
Earlier work this paper cites.
Z. Huang and N. Wang, “Like what you like: Knowledge distill via neuron selectivity transfer,” arXiv preprint arXiv:1707.01219 , 2017
Original
2017
Earlier work this paper cites.
J. Yim, D. Joo, J. Bae, and J. Kim, “A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 4133–4141
2017
Earlier work this paper cites.
S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,” International Conference on Learning Representations , 2017
2017
Earlier work this paper cites.

Similar

Z. Xu, Y.-C. Hsu, and J. Huang, “Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks,” arXiv preprint arXiv:1709.00513 , 2017
Original
2017
Cited alongside, same era.
X. Wang, A. Shrivastava, and A. Gupta, “A-fast-rcnn: Hard positive generation via adversary for object detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 2606–2615
2017
Cited alongside, same era.
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2980–2988
2017
Cited alongside, same era.
H. Chen, Y. Wang, C. Xu, C. Xu, and D. Tao, “Learning student networks via feature embedding,” arXiv preprint arXiv:1812.06597 , 2018
Original
2018
Cited alongside, same era.
H. Wang, H. Zhao, X. Li, and X. Tan, “Progressive blockwise knowledge distillation for neural network acceleration.” in IJCAI , 2018, pp. 2769–2775
2018
Cited alongside, same era.
N. Passalis and A. Tefas, “Unsupervised knowledge transfer using similarity embeddings,” IEEE transactions on neural networks and learning systems , vol. 30, no. 3, pp. 946–950, 2018
2018
Cited alongside, same era.
W.-C. Chen, C.-C. Chang, and C.-R. Lee, “Knowledge distillation with feature maps for image classification,” in Asian Conference on Computer Vision . Springer, 2018, pp. 200–215
2018
Cited alongside, same era.
T. Furlanello, Z. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, “Born again neural networks,” in International Conference on Machine Learning , 2018, pp. 1607–1616
2018
Cited alongside, same era.
M. Gao, Y. Shen, Q. Li, J. Yan, L. Wan, D. Lin, C. C. Loy, and X. Tang, “An embarrassingly simple approach for knowledge distillation,” arXiv preprint arXiv:1812.01819 , 2018
Original
2018
Cited alongside, same era.
C. Yang, L. Xie, S. Qiao, and A. Yuille, “Knowledge distillation in generations: More tolerant teachers educate better students,” arXiv preprint arXiv:1805.05551 , 2018
Original
2018
Cited alongside, same era.
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” International Conference on Learning Representations , 2018
2018
Cited alongside, same era.
H. Bagherinezhad, M. Horton, M. Rastegari, and A. Farhadi, “Label refinery: Improving imagenet classification through label progression,” arXiv preprint arXiv:1805.02641 , 2018
Original
2018
Cited alongside, same era.

Then

B. Heo, M. Lee, S. Yun, and J. Y. Choi, “Knowledge transfer via distillation of activation boundaries formed by hidden neurons,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, 2019, pp. 3779–3787
2019
Closest in time.
W. Park, D. Kim, Y. Lu, and M. Cho, “Relational knowledge distillation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 3967–3976
2019
Closest in time.
F. Tung and G. Mori, “Similarity-preserving knowledge distillation,” in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 1365–1374
2019
Closest in time.
T. Guo, C. Xu, S. He, B. Shi, C. Xu, and D. Tao, “Robust student network learning,” IEEE transactions on neural networks and learning systems , 2019
2019
Closest in time.
S.-I. Mirzadeh, M. Farajtabar, A. Li, and H. Ghasemzadeh, “Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher,” arXiv preprint arXiv:1902.03393 , 2019
Original
2019
Closest in time.
Z. Shen, Z. He, and X. Xue, “Meal: Multi-model ensemble via adversarial learning,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, 2019, pp. 4886–4893
2019
Closest in time.
T.-B. Xu and C.-L. Liu, “Data-distortion guided self-distillation for deep neural networks,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, 2019, pp. 5565–5572
2019
Closest in time.
L. Yuan, F. E. Tay, G. Li, T. Wang, and J. Feng, “Revisit knowledge distillation: a teacher-free framework,” arXiv preprint arXiv:1909.11723 , 2019
Original
2019
Closest in time.
L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma, “Be your own teacher: Improve the performance of convolutional neural networks via self distillation,” in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 3713–3722
2019
Closest in time.
H. Zhao, X. Sun, J. Dong, C. Chen, and Z. Dong, “Highlight every step: Knowledge distillation via collaborative teaching,” arXiv preprint arXiv:1907.09643 , 2019
Original
2019
Closest in time.
M. Yuan and Y. Peng, “Ckd: Cross-task knowledge distillation for text-to-image synthesis,” IEEE Transactions on Multimedia , 2019
2019
Closest in time.
H. Chen, Y. Wang, C. Xu, Z. Yang, C. Liu, B. Shi, C. Xu, C. Xu, and Q. Tian, “Data-free learning of student networks,” in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 3514–3522
2019
Closest in time.
X. Han, X. Song, Y. Yao, X.-S. Xu, and L. Nie, “Neural compatibility modeling with probabilistic knowledge distillation,” IEEE Transactions on Image Processing , vol. 29, pp. 871–882, 2019
2019
Closest in time.
J. Li, K. Fu, S. Zhao, and S. Ge, “Spatiotemporal knowledge distillation for efficient estimation of aerial video saliency,” IEEE Transactions on Image Processing , vol. 29, pp. 1902–1914, 2019
2019
Closest in time.
Q. Ding, S. Wu, H. Sun, J. Guo, and S.-T. Xia, “Adaptive regularization of labels,” arXiv preprint arXiv:1908.05474 , 2019
Original
2019
Closest in time.
B. Li, Y. Liu, and X. Wang, “Gradient harmonized single-stage detector,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, 2019, pp. 8577–8584
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…