PACT: Parameterized Clipping Activation for Quantized Neural Networks

Understand

Deep learning algorithms achieve high classification accuracy at the expense of significant computation cost.

• To address this cost, a number of quantization schemes have been proposed - but most of these techniques focused on quantizing weights, which are relatively smaller in size compared to activations.
• This paper proposes a novel quantization scheme for activations during training - that enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation.
• This technique, PArameterized Clipping acTivation (PACT), uses an activation clipping parameter $\alpha$ that is optimized during training to find the right quantization scale.