RTN: Reparameterized Ternary Network

Understand

To deploy deep neural networks on resource-limited devices, quantization has been widely explored.

• In this work, we study the extremely low-bit networks which have tremendous speed-up, memory saving with quantized activation and weights.
• We first bring up three omitted issues in extremely low-bit networks: the squashing range of quantized values; the gradient vanishing during backpropagation and the unexploited hardware acceleration of ternary networks.
• By reparameterizing quantized activation and weights vector with full precision scale and offset for fixed ternary vector, we decouple the range and magnitude from the direction to extenuate the three issues.