Weight Distillation: Transferring the Knowledge in Neural Network Parameters

Understand

Knowledge distillation has been proven to be effective in model acceleration and compression.

• It allows a small network to learn to generalize in the same way as a large network.
• Recent successes in pre-training suggest the effectiveness of transferring model parameters.
• Inspired by this, we investigate methods of model acceleration and compression in another line of research.