2018

Knowledge Transfer with Jacobian Matching

Srinivas, Suraj, Fleuret, Francois

Understand

Classical distillation methods transfer representations from a "teacher" neural network to a "student" network by matching their output activations.

Recent methods also match the Jacobians, or the gradient of output activations with the input.
However, this involves making some ad hoc decisions, in particular, the choice of the loss function.
In this paper, we first establish an equivalence between Jacobian matching and distillation with input noise, from which we derive appropriate loss functions for Jacobian matching.

Improving generalization performance using double backpropagation
Drucker, Harris and Le Cun, Yann · 1992
Earlier work this paper cites.
Training with noise is equivalent to tikhonov regularization
Bishop, Chris M · 1995
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, Alex and Hinton, Geoffrey · 2009
Earlier work this paper cites.
Recognizing indoor scenes
Quattoni, A and Torralba, A · 2009
Earlier work this paper cites.
A theory of learning from different domains
Ben-David, Shai, Blitzer, John, Crammer, Koby, Kulesza, Alex, Pereira, Fernando, and Vaughan, Jennifer Wortman · 2010
Earlier work this paper cites.
A survey on transfer learning
Pan, Sinno Jialin and Yang, Qiang · 2010
Earlier work this paper cites.
Robustness and generalization
Xu, Huan and Mannor, Shie · 2012
Earlier work this paper cites.
Do deep networks really need to be deep
Ba, LJ and Caruana, R · 2014
Earlier work this paper cites.

Fitnets: Hints for thin deep nets
Original
Romero, Adriana, Ballas, Nicolas, Kahou, Samira Ebrahimi, Chassang, Antoine, Gatta, Carlo, and Bengio, Yoshua · 2014
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, Karen and Zisserman, Andrew · 2014
Cited alongside, same era.
Dropout: A simple way to prevent neural networks from overfitting
Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan · 2014
Cited alongside, same era.
How transferable are features in deep neural networks?
Yosinski, Jason, Clune, Jeff, Bengio, Yoshua, and Lipson, Hod · 2014
Cited alongside, same era.
Distilling the knowledge in a neural network
Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, et al · 2015
Cited alongside, same era.
Active long term memory networks
Original
Furlanello, Tommaso, Zhao, Jiaping, Saxe, Andrew M, Itti, Laurent, and Tjan, Bosco S · 2016
Cited alongside, same era.
Deep residual learning for image recognition
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian · 2016
Cited alongside, same era.

Less-forgetting learning in deep neural networks
Original
Jung, Heechul, Ju, Jeongwoo, Jung, Minju, and Kim, Junmo · 2016
Later among the works it cites.
Learning without forgetting
Li, Zhizhong and Hoiem, Derek · 2016
Later among the works it cites.
Deep model compression: Distilling knowledge from noisy teachers
Original
Sau, Bharat Bhusan and Balasubramanian, Vineeth N · 2016
Later among the works it cites.
Analysis of deep neural networks with extended data jacobian matrix
Wang, Shengjie, Mohamed, Abdel-rahman, Caruana, Rich, Bilmes, Jeff, Plilipose, Matthai, Richardson, Matthew, Geras, Krzysztof, Urban, Gregor, and Aslan, Ozlem · 2016
Later among the works it cites.
Parseval networks: Improving robustness to adversarial examples
Cisse, Moustapha, Bojanowski, Piotr, Grave, Edouard, Dauphin, Yann, and Usunier, Nicolas · 2017
Later among the works it cites.
Sobolev training for neural networks
Czarnecki, Wojciech Marian, Osindero, Simon, Jaderberg, Max, Świrszcz, Grzegorz, and Pascanu, Razvan · 2017
Later among the works it cites.
Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer
Zagoruyko, Sergey and Komodakis, Nikos · 2017
Later among the works it cites.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…