Prune Once for All: Sparse Pre-Trained Language Models

Understand

Transformer-based language models are applied to a wide range of applications in natural language processing.

• However, they are inefficient and difficult to deploy.
• In recent years, many compression algorithms have been proposed to increase the implementation efficiency of large Transformer-based models on target hardware.
• In this work we present a new method for training sparse pre-trained Transformer language models by integrating weight pruning and model distillation.