2020

WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization

Elthakeb, Ahmed T., Pilligundla, Prannoy, Mireshghallah, Fatemehsadat et al.

Understand

As deep neural networks make their ways into different domains, their compute efficiency is becoming a first-order constraint.

Deep quantization, which reduces the bitwidth of the operations (below 8 bits), offers a unique opportunity as it can reduce both the storage and compute requirements of the network super-linearly.
However, if not employed with diligence, this can lead to significant accuracy loss.
Due to the strong inter-dependence between layers and exhibiting different characteristics across the same network, choosing an optimal bitwidth per layer granularity is not a straight forward.

Built on

Learned step size quantization
Original
Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S · 1902
Earlier work this paper cites.
SinReQ: Generalized sinusoidal regularization for low-bitwidth deep quantized training
Original
Elthakeb, A. T., Pilligundla, P., and Esmaeilzadeh, H · 1905
Earlier work this paper cites.
Mixed precision dnns: All you need is a good parametrization
Original
Uhlich, S., Mauch, L., Yoshiyama, K., Cardinaux, F., García, J. A., Tiedemann, S., Kemp, T., and Nakamura, A · 1905
Earlier work this paper cites.
Divide and Conquer: Leveraging intermediate feature representations for quantized training of neural networks
Original
Elthakeb, A. T., Pilligundla, P., and Esmaeilzadeh, H · 1906
Earlier work this paper cites.
Differentiable soft quantization: Bridging full-precision and low-bit neural networks
Original
Gong, R., Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, F., and Yan, J · 1908
Earlier work this paper cites.
Learning internal representations by error propagation
Rumelhart, D. E., Hinton, G. E., and Williams, R. J · 1986
Earlier work this paper cites.
Estimating or propagating gradients through stochastic neurons for conditional computation
Original
Bengio, Y., Léonard, N., and Courville, A. C · 2013
Earlier work this paper cites.
The Loss Surfaces of Multilayer Networks
Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., and LeCun, Y · 2015
Earlier work this paper cites.

Similar

Stripes: Bit-serial deep neural network computing
Judd, P., Albericio, J., Hetherington, T. H., Aamodt, T. M., and Moshovos, A · 2016
Cited alongside, same era.
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Ni, Z., Zhou, X., Wen, H., Wu, Y., and Zou, Y · 2016
Cited alongside, same era.
Loss-aware binarization of deep networks
Hou, L., Yao, Q., and Kwok, J. T · 2017
Cited alongside, same era.
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y · 2017
Cited alongside, same era.
Trained Ternary Quantization
Zhu, C., Han, S., Mao, H., and Dally, W. J · 2017
Cited alongside, same era.
Variational network quantization
Achterhold, J., Köhler, J. M., Schmeink, A., and Genewein, T · 2018
Cited alongside, same era.
ReLeQ: A reinforcement learning approach for deep quantization of neural networks
Original
Elthakeb, A. T., Pilligundla, P., Mireshghallah, F., Yazdanbakhsh, A., and Esmaeilzadeh, H · 2018
Cited alongside, same era.
Loss-aware weight quantization of deep networks
Hou, L. and Kwok, J. T · 2018
Cited alongside, same era.
Pact: Parameterized clipping activation for quantized neural networks
Original
Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srinivasan, V., and Gopalakrishnan, K
Cited in the paper.
Learning low precision deep neural networks through regularization
Original
Choi, Y., El-Khamy, M., and Lee, J
Cited in the paper.
Stripes: Bit-serial deep neural network computing
Judd, P., Albericio, J., Hetherington, T. H., Aamodt, T. M., and Moshovos, A
Cited in the paper.

Then

Visualizing the Loss Landscape of Neural Nets
Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T · 2018
Later among the works it cites.
WRPN: Wide Reduced-Precision Networks
Mishra, A. K., Nurvitadhi, E., Cook, J. J., and Marr, D · 2018
Later among the works it cites.
On periodic functions as regularizers for quantization of neural networks
Original
Naumov, M., Diril, U., Park, J., Ray, B., Jablonski, J., and Tulloch, A · 2018
Later among the works it cites.
Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network
Sharma, H., Park, J., Suda, N., Lai, L., Chau, B., Chandra, V., and Esmaeilzadeh, H · 2018
Later among the works it cites.
A unified framework of dnn weight pruning and weight clustering/quantization using admm
Original
Ye, S., Zhang, T., Zhang, K., Li, J., Xie, J., Liang, Y., Liu, S., Lin, X., and Wang, Y · 2018
Later among the works it cites.
Lq-nets: Learned quantization for highly accurate and compact deep neural networks
Zhang, D., Yang, J., Ye, D., and Hua, G · 2018
Later among the works it cites.
Neural network distiller, June 2018
Zmora, N., Jacob, G., and Novik, G · 2018
Later among the works it cites.
HAQ: hardware-aware automated quantization with mixed precision
Wang, K., Liu, Z., Lin, Y., Lin, J., and Han, S · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…