A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off

Understand

Reducing the precision of weights and activation functions in neural network training, with minimal impact on performance, is essential for the deployment of these models in resource-constrained environments.

• We apply mean-field techniques to networks with quantized activations in order to evaluate the degree to which quantization degrades signal propagation at initialization.
• We derive initialization schemes which maximize signal propagation in such networks and suggest why this is helpful for generalization.
• Building on these results, we obtain a closed form implicit equation for $L_{\max}$, the maximal trainable depth (and hence model capacity), given $N$, the number of quantization levels in the activation function.