FEED: Feature-level Ensemble for Knowledge Distillation

Understand

Knowledge Distillation (KD) aims to transfer knowledge in a teacher-student framework, by providing the predictions of the teacher network to the student network in the training stage to help the student network generalize better.

• It can use either a teacher with high capacity or {an} ensemble of multiple teachers.
• However, the latter is not convenient when one wants to use feature-map-based distillation methods.
• For a solution, this paper proposes a versatile and powerful training algorithm named FEature-level Ensemble for knowledge Distillation (FEED), which aims to transfer the ensemble knowledge using multiple teacher networks.