Pruning Attention Heads of Transformer Models Using A* Search: A Novel Approach to Compress Big NLP Architectures

Understand

Recent years have seen a growing adoption of Transformer models such as BERT in Natural Language Processing and even in Computer Vision.

• However, due to their size, there has been limited adoption of such models within resource-constrained computing environments.
• This paper proposes novel pruning algorithm to compress transformer models by eliminating redundant Attention Heads.
• We apply the A* search algorithm to obtain a pruned model with strict accuracy guarantees.