KTAN: Knowledge Transfer Adversarial Network

Understand

To reduce the large computation and storage cost of a deep convolutional neural network, the knowledge distillation based methods have pioneered to transfer the generalization ability of a large (teacher) deep network to a light-weight (student) network.

• However, these methods mostly focus on transferring the probability distribution of the softmax layer in a teacher network and thus neglect the intermediate representations.
• In this paper, we propose a knowledge transfer adversarial network to better train a student network.
• Our technique holistically considers both intermediate representations and probability distributions of a teacher network.