BinaryBERT: Pushing the Limit of BERT Quantization

Understand

The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution.

• In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization.
• We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape.
• Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network.