TernaryBERT: Distillation-aware Ultra-low Bit BERT

Understand

Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks.However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices.

• In this work, we propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model.
• Specifically, we use both approximation-based and loss-aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT.
• Moreover, to reduce the accuracy degradation caused by the lower capacity of low bits, we leverage the knowledge distillation technique in the training process.