EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation

Understand

Pre-trained language models have shown remarkable results on various NLP tasks.

• Nevertheless, due to their bulky size and slow inference speed, it is hard to deploy them on edge devices.
• In this paper, we have a critical insight that improving the feed-forward network (FFN) in BERT has a higher gain than improving the multi-head attention (MHA) since the computational cost of FFN is 2$\sim$3 times larger than MHA.
• Hence, to compact BERT, we are devoted to designing efficient FFN as opposed to previous works that pay attention to MHA.