Online Knowledge Distillation with Diverse Peers

Understand

Distillation is an effective knowledge-transfer technique that uses predicted distributions of a powerful teacher model as soft targets to train a less-parameterized student model.

• A pre-trained high capacity teacher, however, is not always available.
• Recently proposed online variants use the aggregated intermediate predictions of multiple student models as targets to train each student model.
• Although group-derived targets give a good recipe for teacher-free distillation, group members are homogenized quickly with simple aggregation functions, leading to early saturated solutions.