Fetching the paper…

Reading the bibliography…

2020

BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization

Nikolić, Miloš, Hacene, Ghouthi Boukli, Bannon, Ciaran et al.

Original
PDF
Post on X

Understand

Neural networks have demonstrably achieved state-of-the art accuracy using low-bitlength integer quantization, yielding both execution time and energy benefits on existing hardware designs that support short bitlengths.

However, the question of finding the minimum bitlength for a desired accuracy remains open.
We introduce a training method for minimizing inference bitlength at any granularity while maintaining accuracy.
Namely, we propose a regularizer that penalizes large bitlength representations throughout the architecture and show how it can be modified to minimize other quantifiable criteria, such as number of operations or memory footprint.

BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization · Around

Built on

S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha, “Learned step size quantization,” CoRR , vol. abs/1902.08153, 2019. [Online]. Available: http://arxiv.org/abs/1902.08153
Original
1902
Earlier work this paper cites.
S. R. Jain, A. Gural, M. Wu, and C. Dick, “Trained uniform quantization for accurate and efficient neural network inference on fixed-point hardware,” CoRR , vol. abs/1903.08066, 2019. [Online]. Available: http://arxiv.org/abs/1903.08066
Original
1903
Earlier work this paper cites.
S. Uhlich, L. Mauch, K. Yoshiyama, F. Cardinaux, J. A. García, S. Tiedemann, T. Kemp, and A. Nakamura, “Differentiable quantization of deep neural networks,” CoRR , vol. abs/1905.11452, 2019. [Online]. Available: http://arxiv.org/abs/1905.11452
Original
1905
Earlier work this paper cites.
A. Krizhevsky, “Learning multiple layers of features from tiny images,” Tech. Rep., 2009
2009
Earlier work this paper cites.
H. Esmaeilzadeh, E. Blem, R. St. Amant, K. Sankaralingam, and D. Burger, “Dark silicon and the end of multicore scaling,” in Proceedings of the 38th Annual International Symposium on Computer Architecture , ser. ISCA ’11. New York, NY, USA: ACM, 2011, pp. 365–376
2011
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep cnns,” in NIPS 25 , F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1097–1105
2012
Earlier work this paper cites.
Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432 , 2013
Original
2013
Earlier work this paper cites.
M. Horowitz, “Computing’s energy problem (and what we can do about it),” IEEE Intl’ Solid-State Circuits Conf. , vol. 57, pp. 10–14, 02 2014
2014
Earlier work this paper cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” arXiv:1409.0575 [cs] , Sep. 2014, arXiv: 1409.0575
Original
2014
Earlier work this paper cites.
M. Courbariaux, Y. Bengio, and J.-P. David, “BinaryConnect: Training Deep Neural Networks with binary weights during propagations,” ArXiv e-prints , Nov. 2015
2015
Earlier work this paper cites.
P. Judd, J. Albericio, T. Hetherington, T. Aamodt, N. Enright Jerger, R. Urtasun, and A. Moshovos, “Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets, arXiv:1511.05236v4 [cs.LG] ,” arXiv.org , 2015
Original
2015
Earlier work this paper cites.
P. Judd, J. Albericio, T. Hetherington, T. Aamodt, and A. Moshovos, “Stripes: Bit-serial Deep Neural Network Computing ,” in Proceedings of the 49th Annual IEEE/ACM International Symposium on Microarchitecture , ser. MICRO-49, 2016
2016
Earlier work this paper cites.
P. Judd, J. Albericio, T. Hetherington, T. M. Aamodt, N. E. Jerger, and A. Moshovos, “Proteus: Exploiting numerical precision variability in deep neural networks,” in Proceedings of the 2016 International Conference on Supercomputing , ser. ICS ’16. New York, NY, USA: ACM, 2016, pp. 23:1–23:12. [Online]. Available: http://doi.acm.org/10.1145/2925426.2926294
2016
Earlier work this paper cites.
P. Judd, J. Albericio, T. Hetherington, T. M. Aamodt, N. E. Jerger, and A. Moshovos, “Proteus: Exploiting numerical precision variability in deep neural networks,” in Proceedings of the 2016 International Conference on Supercomputing , ser. ICS ’16. New York, NY, USA: ACM, 2016, pp. 23:1–23:12. [Online]. Available: http://doi.acm.org/10.1145/2925426.2926294
2016
Earlier work this paper cites.
I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks,” in Advances in neural information processing systems , 2016, pp. 4107–4115
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.

Similar

G. Huang, Z. Liu, and K. Q. Weinberger, “Densely connected convolutional networks,” CoRR , vol. abs/1608.06993, 2016. [Online]. Available: http://arxiv.org/abs/1608.06993
Original
2016
Cited alongside, same era.
S. Xie, R. B. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” CoRR , vol. abs/1611.05431, 2016. [Online]. Available: http://arxiv.org/abs/1611.05431
Original
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” CoRR , vol. abs/1603.05027, 2016. [Online]. Available: http://arxiv.org/abs/1603.05027
Original
2016
Cited alongside, same era.

Then

Y. Umuroglu, L. Rasnayake, and M. Själander, “BISMO: A scalable bit-serial matrix multiplication overlay for reconfigurable computing,” CoRR , vol. abs/1806.08862, 2018. [Online]. Available: http://arxiv.org/abs/1806.08862
Original
2018
Later among the works it cites.
J. Choi, Z. Wang, S. Venkataramani, P. I. Chuang, V. Srinivasan, and K. Gopalakrishnan, “PACT: parameterized clipping activation for quantized neural networks,” CoRR , vol. abs/1805.06085, 2018. [Online]. Available: http://arxiv.org/abs/1805.06085
Original
2018
Later among the works it cites.
J. Fromm, S. Patel, and M. Philipose, “Heterogeneous bitwidth binarization in convolutional neural networks,” CoRR , vol. abs/1805.10368, 2018. [Online]. Available: http://arxiv.org/abs/1805.10368
Original
2018
Later among the works it cites.
E. Park, S. Yoo, and P. Vajda, “Value-aware quantization for training and inference of neural networks,” CoRR , vol. abs/1804.07802, 2018. [Online]. Available: http://arxiv.org/abs/1804.07802
Original
2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers, R. Boyle, P.-l. Cantin, C. Chao, C. Clark, J. Coriell, M. Daley, M. Dau, J. Dean, B. Gelb, T. V. Ghaemmaghami, R. Gottipati, W. Gulland, R. Hagmann, C. R. Ho, D. Hogberg, J. Hu, R. Hundt, D. Hurt, J. Ibarz, A. Jaffey, A. Jaworski, A. Kaplan, H. Khaitan, D. Killebrew, A. Koch, N. Kumar, S. Lacy, J. Laudon, J. Law, D. Le, C. Leary, Z. Liu, K. Lucke, A. Lundin, G. MacKean, A. Maggiore, M. Mahony, K. Miller, R. Nagarajan, R. Narayanaswami, R. Ni, K. Nix, T. Norrie, M. Omernick, N. Penukonda, A. Phelps, J. Ross, M. Ross, A. Salek, E. Samadiani, C. Severn, G. Sizikov, M. Snelham, J. Souter, D. Steinberg, A. Swing, M. Tan, G. Thorson, B. Tian, H. Toma, E. Tuttle, V. Vasudevan, R. Walter, W. Wang, E. Wilcox, and D. H. Yoon, “In-datacenter performance analysis of a tensor processing unit,” in Proceedings of the 44th Annual International Symposium on Computer Architecture , ser. ISCA ’17, 2017, pp. 1–12

2017

Cited alongside, same era.

H. Sharma, J. Park, N. Suda, L. Lai, B. Chau, J. K. Kim, V. Chandra, and H. Esmaeilzadeh, “Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural networks,” CoRR , vol. abs/1712.01507, 2017. [Online]. Available: http://arxiv.org/abs/1712.01507

2017

Cited alongside, same era.

J. Albericio, A. Delmás, P. Judd, S. Sharify, G. O’Leary, R. Genov, and A. Moshovos, “Bit-pragmatic deep neural network computing,” in Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture , ser. MICRO-50 ’17, 2017, pp. 382–394

2017

Cited alongside, same era.

A. D. Lascorz, S. Sharify, P. Judd, K. Siu, M. Nikolic, and A. Moshovos, “Dpred: Making typical activation values matter in deep learning computing,” CoRR , vol. abs/1804.06732, 2017. [Online]. Available: http://arxiv.org/abs/1804.06732

2017

Cited alongside, same era.

S. Sharify, A. D. Lascorz, P. Judd, and A. Moshovos, “Loom: Exploiting weight and activation precisions to accelerate convolutional neural networks,” CoRR , vol. abs/1706.07853, 2017. [Online]. Available: http://arxiv.org/abs/1706.07853

2017

Cited alongside, same era.

A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen, “Incremental network quantization: Towards lossless cnns with low-precision weights,” 02 2017

2017

Cited alongside, same era.

A. Delmas, P. Judd, S. Sharify, and A. Moshovos, “Dynamic stripes: Exploiting the dynamic precision requirements of activation values in neural networks,” CoRR , vol. abs/1706.00504, 2017. [Online]. Available: http://arxiv.org/abs/1706.00504

2017

Cited alongside, same era.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017

2017

Cited alongside, same era.

Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, and J. Feng, “Dual path networks,” CoRR , vol. abs/1707.01629, 2017. [Online]. Available: http://arxiv.org/abs/1707.01629

2017

Cited alongside, same era.

A. T. Elthakeb, P. Pilligundla, A. Yazdanbakhsh, S. Kinzer, and H. Esmaeilzadeh, “Releq: A reinforcement learning approach for deep quantization of neural networks,” CoRR , vol. abs/1811.01704, 2018. [Online]. Available: http://arxiv.org/abs/1811.01704

2018

Later among the works it cites.

K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, “HAQ: hardware-aware automated quantization,” CoRR , vol. abs/1811.08886, 2018. [Online]. Available: http://arxiv.org/abs/1811.08886

2018

Later among the works it cites.

J. Lee, C. Kim, S. Kang, D. Shin, S. Kim, and H.-J. Yoo, “Unpu: A 50.6tops/w unified deep neural network accelerator with 1b-to-16b fully-variable weight bit-precision,” 2018 IEEE International Solid - State Circuits Conference - (ISSCC) , pp. 218–220, 2018

2018

Later among the works it cites.

J. Howard et al. , “fastai,” https://github.com/fastai/fastai , 2018

2018

Later among the works it cites.

M. Sandler, A. F. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 4510–4520, 2018

2018

Later among the works it cites.

A. Mishra, E. Nurvitadhi, J. J. Cook, and D. Marr, “WRPN: Wide reduced-precision networks,” in International Conference on Learning Representations , 2018. [Online]. Available: https://openreview.net/forum?id=B1ZvaaeAZ

2018

Later among the works it cites.

H. Wu, “Low precision inference on gpu,” Nvidia, Tech. Rep., 2019

2019

Later among the works it cites.

A. D. Lascorz, S. Sharify, I. Edo, D. M. Stuart, O. M. Awad, P. Judd, M. Mahmoud, M. Nikolic, K. Siu, Z. Poulos, and A. Moshovos, “Shapeshifter: Enabling fine-grain data width adaptation in deep learning,” in Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture , ser. MICRO ’52. New York, NY, USA: Association for Computing Machinery, 2019, p. 28–41. [Online]. Available: https://doi.org/10.1145/3352460.3358295

2019

Later among the works it cites.

O. Bilaniuk, S. Wagner, Y. Savaria, and J.-P. David, “Bit-slicing fpga accelerator for quantized neural networks,” in 2019 IEEE International Symposium on Circuits and Systems (ISCAS) . IEEE, 2019, pp. 1–5

2019

Later among the works it cites.

M. Nikolic, M. Mahmoud, Y. Zhao, R. Mullins, and A. Moshovos, “Characterizing sources of ineffectual computations in deep learning networks,” in International Symposium on Performance Analysis of Systems and Software , 03 2019

2019

Later among the works it cites.

A. Delmas Lascorz, P. Judd, D. M. Stuart, Z. Poulos, M. Mahmoud, S. Sharify, M. Nikolic, K. Siu, and A. Moshovos, “Bit-tactical: A software/hardware approach to exploiting value and bit sparsity in neural networks,” in Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems , ser. ASPLOS ’19. New York, NY, USA: ACM, 2019, pp. 749–763. [Online]. Available: http://doi.acm.org/10.1145/3297858.3304041

2019

Later among the works it cites.