Fetching the paper…
Reading the bibliography…
Knowledge distillation (KD) is a very popular method for model size reduction.
“Model compression,”
Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil, · 2006
Earlier work this paper cites.
“Fixed-point feedforward deep neural network design using weights +1, 0, and -1,”
Kyuyeon Hwang and Wonyong Sung, · 2014
Earlier work this paper cites.
“Fitnets: Hints for thin deep nets,”
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio, · 2014
Earlier work this paper cites.
“Distilling the knowledge in a neural network,”
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, · 2015
Earlier work this paper cites.
“Resiliency of deep neural networks under quantization,”
Wonyong Sung, Sungho Shin, and Kyuyeon Hwang, · 2015
Earlier work this paper cites.
“Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,”
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou, · 2016
Earlier work this paper cites.
“Recurrent neural network training with dark knowledge transfer,”
Zhiyuan Tang, Dong Wang, and Zhiyong Zhang, · 2016
Earlier work this paper cites.
“Xnor-net: Imagenet classification using binary convolutional neural networks,”
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi, · 2016
Earlier work this paper cites.
Sergey Zagoruyko and Nikos Komodakis, · 2016
Earlier work this paper cites.
“Deep residual learning for image recognition,”
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, · 2016
Cited alongside, same era.
“Quantized neural networks: Training neural networks with low precision weights and activations.,”
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio, · 2017
Cited alongside, same era.
“Balanced quantization: An effective and efficient approach to quantized neural networks,”
Shu-Chang Zhou, Yu-Zhi Wang, He Wen, Qin-Yao He, and Yu-Heng Zou, · 2017
Cited alongside, same era.
“Domain adaptation of dnn acoustic models using knowledge distillation,”
Taichi Asami, Ryo Masumura, Yoshikazu Yamaguchi, Hirokazu Masataki, and Yushi Aono, · 2017
Cited alongside, same era.
“Knowledge distillation using unlabeled mismatched images,”
Mandar Kulkarni, Kalpesh Patil, and Shirish Karande, · 2017
Cited alongside, same era.
“Neural compatibility modeling with attentive knowledge distillation,”
Xuemeng Song, Fuli Feng, Xianjing Han, Xin Yang, Wei Liu, and Liqiang Nie, · 2018
Later among the works it cites.
“Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy,”
Asit Mishra and Debbie Marr, · 2018
Later among the works it cites.
“Model compression via distillation and quantization,”
Antonio Polino, Razvan Pascanu, and Dan Alistarh, · 2018
Later among the works it cites.
“Towards effective low-bitwidth convolutional neural networks,”
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid, · 2018
Later among the works it cites.
“Deepvid: Deep visual interpretation and diagnosis for image classifiers via knowledge distillation,”
Junpeng Wang, Liang Gou, Wei Zhang, Hao Yang, and Han-Wei Shen, · 2019
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
“A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,”
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim, · 2017
Cited alongside, same era.
“Fixed-point optimization of deep neural networks with adaptive step size retraining,”
Sungho Shin, Yoonho Boo, and Wonyong Sung, · 2017
Cited alongside, same era.
“Alternating multi-bit quantization for recurrent neural networks,”
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha, · 2018
Cited alongside, same era.
“Relational knowledge distillation,”
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho, · 2019
Closest in time.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh, · 2019
Closest in time.
“Memorization capacity of deep neural networks under parameter quantization,”
Yoonho Boo, Sungho Shin, and Wonyong Sung, · 2019
Closest in time.