ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression

Understand

Pretrained language models (PLMs) such as BERT adopt a training paradigm which first pretrain the model in general data and then finetune the model on task-specific data, and have recently achieved great success.

• However, PLMs are notorious for their enormous parameters and hard to be deployed on real-life applications.
• Knowledge distillation has been prevailing to address this problem by transferring knowledge from a large teacher to a much smaller student over a set of data.
• We argue that the selection of thee three key components, namely teacher, training data, and learning objective, is crucial to the effectiveness of distillation.