Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks

Understand

The deep layers of modern neural networks extract a rather rich set of features as an input propagates through the network.

• This paper sets out to harvest these rich intermediate representations for quantization with minimal accuracy loss while significantly reducing the memory footprint and compute intensity of the DNN.
• This paper utilizes knowledge distillation through teacher-student paradigm (Hinton et al., 2015) in a novel setting that exploits the feature extraction capability of DNNs for higher-accuracy quantization.
• As such, our algorithm logically divides a pretrained full-precision DNN to multiple sections, each of which exposes intermediate features to train a team of students independently in the quantized domain.