Fetching the paper…
Reading the bibliography…
In this paper, we present a thorough evaluation of the efficacy of knowledge distillation and its dependence on student and teacher architectures.
Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping
Rich Caruana, Steve Lawrence, and C Lee Giles · 2001
Earlier work this paper cites.
Model compression
Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil · 2006
Earlier work this paper cites.
Do deep nets really need to be deep?
Jimmy Ba and Rich Caruana · 2014
Earlier work this paper cites.
Distilling the knowledge in a neural network
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean · 2015
Earlier work this paper cites.
Distilling knowledge from ensembles of neural networks for speech recognition
Yevgen Chebotar and Austin Waters · 2016
Earlier work this paper cites.
Cross modal distillation for supervision transfer
Saurabh Gupta, Judy Hoffman, and Jitendra Malik · 2016
Earlier work this paper cites.
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Song Han, Huizi Mao, and William J Dally · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Sequence-level knowledge distillation
Yoon Kim and Alexander M. Rush · 2016
Earlier work this paper cites.
Quantized convolutional neural networks for mobile devices
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng · 2016
Cited alongside, same era.
Wide residual networks
Sergey Zagoruyko and Nikos Komodakis · 2016
Cited alongside, same era.
Learning efficient object detection models with knowledge distillation
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker · 2017
Cited alongside, same era.
On calibration of modern neural networks
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger · 2017
Cited alongside, same era.
Densely connected convolutional networks
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger · 2017
Cited alongside, same era.
Like what you like: Knowledge distill via neuron selectivity transfer
Zehao Huang and Naiyan Wang · 2017
A gift from knowledge distillation: Fast optimization, network minimization and transfer learning
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim · 2017
Later among the works it cites.
Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer
Sergey Zagoruyko and Nikos Komodakis · 2017
Later among the works it cites.
Born again neural networks
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar · 2018
Later among the works it cites.
Morphnet: Fast and simple resource-constrained structure learning of deep networks
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi · 2018
Later among the works it cites.
Knowledge distillation by on-the-fly native ensemble
Xu Lan, Xiatian Zhu, and Shaogang Gong · 2018
Later among the works it cites.
Adversarial teacher-student learning for unsupervised domain adaptation
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Like what you like: Knowledge distill via neuron selectivity transfer
Zehao Huang and Naiyan Wang · 2017
Cited alongside, same era.
Early stopping without a validation set
Maren Mahsereci, Lukas Balles, Christoph Lassner, and Philipp Hennig · 2017
Cited alongside, same era.
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
Antti Tarvainen and Harri Valpola · 2017
Cited alongside, same era.
Zhong Meng, Jinyu Li, Yifan Gong, and Biing-Hwang Juang · 2018
Later among the works it cites.
Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy
Asit Mishra and Debbie Marr · 2018
Later among the works it cites.
Knowledge distillation in generations: More tolerant teachers educate better students
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan Yuille · 2018
Later among the works it cites.
Deep mutual learning
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu · 2018
Later among the works it cites.