Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students

Understand

We focus on the problem of training a deep neural network in generations.

• The flowchart is that, in order to optimize the target network (student), another network (teacher) with the same architecture is first trained, and used to provide part of supervision signals in the next stage.
• While this strategy leads to a higher accuracy, many aspects (e.g., why teacher-student optimization helps) still need further explorations.
• This paper studies this problem from a perspective of controlling the strictness in training the teacher network.