OverQ: Opportunistic Outlier Quantization for Neural Network Accelerators

Understand

Outliers in weights and activations pose a key challenge for fixed-point quantization of neural networks.

• While they can be addressed by fine-tuning, this is not practical for ML service providers (e.g., Google or Microsoft) who often receive customer models without training data.
• Specialized hardware for handling activation outliers can enable low-precision neural networks, but at the cost of nontrivial area overhead.
• We instead propose overwrite quantization (OverQ), a lightweight hardware technique that opportunistically increases bitwidth for activation outliers by overwriting nearby zeros.