LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression

Understand

BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks.

• However, a major blocking issue of applying BERT to online services is that it is memory-intensive and leads to unsatisfactory latency of user requests, raising the necessity of model compression.
• Existing solutions leverage the knowledge distillation framework to learn a smaller model that imitates the behaviors of BERT.
• However, the training procedure of knowledge distillation is expensive itself as it requires sufficient training data to imitate the teacher model.