Learning Student Networks via Feature Embedding

Understand

Deep convolutional neural networks have been widely used in numerous applications, but their demanding storage and computational resource requirements prevent their applications on mobile devices.

• Knowledge distillation aims to optimize a portable student network by taking the knowledge from a well-trained heavy teacher network.
• Traditional teacher-student based methods used to rely on additional fully-connected layers to bridge intermediate layers of teacher and student networks, which brings in a large number of auxiliary parameters.
• In contrast, this paper aims to propagate information from teacher to student without introducing new variables which need to be optimized.