2018

Learning Sparse Low-Precision Neural Networks With Learnable Regularization

Choi, Yoojin, El-Khamy, Mostafa, Lee, Jungwon

Understand

We consider learning deep neural networks (DNNs) that consist of low-precision weights and activations for efficient inference of fixed-point operations.

In training low-precision networks, gradient descent in the backward pass is performed with high-precision weights while quantized low-precision weights and activations are used in the forward pass to calculate the loss function for training.
Thus, the gradient descent becomes suboptimal, and accuracy loss follows.
In order to reduce the mismatch in the forward and backward passes, we utilize mean squared quantization error (MSQE) regularization.

Built on

J.-J. Moreau, “Proximité et dualité dans un espace hilbertien,” Bulletin de la Société mathématique de France , vol. 93, pp. 273–299, 1965
1965
Earlier work this paper cites.
K. Yosida, Functional Analysis . Springer–Verlag, 1965
1965
Earlier work this paper cites.
S. J. Nowlan and G. E. Hinton, “Simplifying neural networks by soft weight-sharing,” Neural Computation , vol. 4, no. 4, pp. 473–493, 1992
1992
Earlier work this paper cites.
J. Seward, “bzip2,” 1998. [Online]. Available: http://www.bzip.org
1998
Earlier work this paper cites.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing , vol. 13, no. 4, pp. 600–612, 2004
2004
Earlier work this paper cites.
C. M. Bishop, Pattern Recognition and Machine Learning . Springer, 2006
2006
Earlier work this paper cites.
R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in International Conference on Curves and Surfaces . Springer, 2010, pp. 711–730
2010
Earlier work this paper cites.
T. M. Cover and J. A. Thomas, Elements of Information Theory . John Wiley & Sons, 2012
2012
Earlier work this paper cites.
A. Gersho and R. M. Gray, Vector Quantization and Signal Compression . Springer Science & Business Media, 2012, vol. 159
2012
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems , 2012, pp. 1097–1105
2012
Earlier work this paper cites.
Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432 , 2013
Original
2013
Earlier work this paper cites.
M. Horowitz, “1.1 computing’s energy problem (and what we can do about it),” in IEEE International Solid-State Circuits Conference , 2014, pp. 10–14
2014
Earlier work this paper cites.
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature , vol. 521, no. 7553, pp. 436–444, 2015
2015
Earlier work this paper cites.
M. Courbariaux, Y. Bengio, and J.-P. David, “BinaryConnect: Training deep neural networks with binary weights during propagations,” in Advances in Neural Information Processing Systems , 2015, pp. 3123–3131
2015
Earlier work this paper cites.
S. Anwar, K. Hwang, and W. Sung, “Fixed point optimization of deep convolutional neural networks for object recognition,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2015, pp. 1131–1135
2015
Earlier work this paper cites.
S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in Advances in Neural Information Processing Systems , 2015, pp. 1135–1143
2015
Earlier work this paper cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision , vol. 115, no. 3, pp. 211–252, 2015
2015
Earlier work this paper cites.
D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations , 2015
2015
Earlier work this paper cites.
Z. Lin, M. Courbariaux, R. Memisevic, and Y. Bengio, “Neural networks with few multiplications,” in International Conference on Learning Representations , 2016
2016
Earlier work this paper cites.
I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks,” in Advances in Neural Information Processing Systems , 2016, pp. 4107–4115
2016
Earlier work this paper cites.

Similar

M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “XNOR-Net: Imagenet classification using binary convolutional neural networks,” in European Conference on Computer Vision . Springer, 2016, pp. 525–542
2016
Cited alongside, same era.
S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou, “DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” arXiv preprint arXiv:1606.06160 , 2016
Original
2016
Cited alongside, same era.
P. L. Combettes and J.-C. Pesquet, “Stochastic approximations and perturbations in forward-backward splitting for monotone operators,” Pure and Applied Functional Analysis , vol. 1, no. 1, pp. 13–37, 2016
2016
Cited alongside, same era.
V. Lebedev and V. Lempitsky, “Fast convnets using group-wise brain damage,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 2554–2564
2016
Cited alongside, same era.
Y. Guo, A. Yao, and Y. Chen, “Dynamic network surgery for efficient DNNs,” in Advances In Neural Information Processing Systems , 2016, pp. 1379–1387
2016
Cited alongside, same era.
S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding,” in International Conference on Learning Representations , 2016
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 770–778
2016
Cited alongside, same era.
C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 38, no. 2, pp. 295–307, 2016
2016
Cited alongside, same era.
V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient processing of deep neural networks: A tutorial and survey,” Proceedings of the IEEE , vol. 105, no. 12, pp. 2295–2329, 2017
2017
Cited alongside, same era.
C. Zhu, S. Han, H. Mao, and W. J. Dally, “Trained ternary quantization,” in International Conference on Learning Representations , 2017
2017
Cited alongside, same era.
Z. Cai, X. He, J. Sun, and N. Vasconcelos, “Deep learning with low precision by half-wave Gaussian quantization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 5918–5926
2017
Cited alongside, same era.
L. Hou, Q. Yao, and J. T. Kwok, “Loss-aware binarization of deep networks,” in International Conference on Learning Representations , 2017
2017
Cited alongside, same era.

Then

Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “Model compression and acceleration for deep neural networks: The principles, progress, and challenges,” IEEE Signal Processing Magazine , vol. 35, no. 1, pp. 126–136, 2018
2018
Closest in time.
L. Hou and J. T. Kwok, “Loss-aware weight quantization of deep networks,” in International Conference on Learning Representations , 2018
2018
Closest in time.
P. Gysel, J. Pimentel, M. Motamedi, and S. Ghiasi, “Ristretto: A framework for empirical study of resource-efficient inference in convolutional neural networks,” IEEE Transactions on Neural Networks and Learning Systems , no. 99, pp. 1–6, 2018
2018
Closest in time.
A. Zhou, A. Yao, K. Wang, and Y. Chen, “Explicit loss-error-aware quantization for low-bit deep neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 9426–9435
2018
Closest in time.
P. Yin, S. Zhang, J. Lyu, S. Osher, Y. Qi, and J. Xin, “BinaryRelax: A relaxation approach for training deep neural networks with quantized weights,” SIAM Journal on Imaging Sciences , vol. 11, no. 4, pp. 2205–2223, 2018
2018
Closest in time.
J. Choi, Z. Wang, S. Venkataramani, P. I.-J. Chuang, V. Srinivasan, and K. Gopalakrishnan, “PACT: Parameterized clipping activation for quantized neural networks,” arXiv preprint arXiv:1805.06085 , 2018
Original
2018
Closest in time.
D. Zhang, J. Yang, D. Ye, and G. Hua, “LQ-Nets: Learned quantization for highly accurate and compact deep neural networks,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 365–382
2018
Closest in time.
J. Faraone, N. Fraser, M. Blott, and P. H. Leong, “SYQ: Learning symmetric quantization for efficient deep neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 4300–4309
2018
Closest in time.
X. Zhang, X. Zhou, M. Lin, and J. Sun, “ShuffleNet: An extremely efficient convolutional neural network for mobile devices,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 6848–6856
2018
Closest in time.
F. Tung and G. Mori, “Deep neural network compression by in-parallel pruning-quantization,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2018
2018
Closest in time.
Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, “AMC: AutoML for model compression and acceleration on mobile devices,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 784–800
2018
Closest in time.
H. Ren, M. El-Khamy, and J. Lee, “CT-SRCNN: Cascade trained and trimmed deep convolutional neural networks for image super resolution,” in Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV) , 2018
2018
Closest in time.
P. Yin, S. Zhang, J. Lyu, S. Osher, Y. Qi, and J. Xin, “Blended coarse gradient descent for full quantization of deep neural networks,” Research in the Mathematical Sciences , vol. 6, no. 1, p. 14, 2019
2019
Closest in time.
Z.-G. Liu and M. Mattina, “Learning low-precision neural networks without straight-through estimator (STE),” in Proceedings of the International Joint Conference on Artificial Intelligence , 2019, pp. 3066–3072
2019
Closest in time.
J. Yang, X. Shen, J. Xing, X. Tian, H. Li, B. Deng, J. Huang, and X.-s. Hua, “Quantization networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 7308–7316
2019
Closest in time.
R. Gong, X. Liu, S. Jiang, T. Li, P. Hu, J. Lin, F. Yu, and J. Yan, “Differentiable soft quantization: Bridging full-precision and low-bit neural networks,” in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 4852–4861
2019
Closest in time.
C. Louizos, M. Reisser, T. Blankevoort, E. Gavves, and M. Welling, “Relaxed quantization for discretized neural networks,” in International Conference on Learning Representations , 2019
2019
Closest in time.
K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, “HAQ: Hardware-aware automated quantization with mixed precision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 8612–8620
2019
Closest in time.
Y. Choi, M. El-Khamy, and J. Lee, “Universal deep neural network compression,” IEEE Journal of Selected Topics in Signal Processing , 2020
2020
Closest in time.
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured sparsity in deep neural networks,” in Advances in Neural Information Processing Systems , 2016, pp. 2074–2082
2082
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…