One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers

Understand

Pre-trained language models (PLMs) achieve great success in NLP.

• However, their huge model sizes hinder their applications in many practical systems.
• Knowledge distillation is a popular technique to compress PLMs, which learns a small student model from a large teacher PLM.
• However, the knowledge learned from a single teacher may be limited and even biased, resulting in low-quality student model.