Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge

Understand

Mutual knowledge distillation (MKD) improves a model by distilling knowledge from another model.

• However, \textit{not all knowledge is certain and correct}, especially under adverse conditions.
• For example, label noise usually leads to less reliable models due to undesired memorization \cite{zhang2017understanding,arpit2017closer}.
• Wrong knowledge misleads the learning rather than helps.