2021

BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization

Yang, Huanrui, Duan, Lin, Chen, Yiran et al.

Understand

Mixed-precision quantization can potentially achieve the optimal tradeoff between performance and compression rate of deep neural networks, and thus, have been widely investigated.

However, it lacks a systematic method to determine the exact quantization scheme.
Previous methods either examine only a small manually-designed search space or utilize a cumbersome neural architecture search to explore the vast search space.
These approaches cannot lead to an optimal quantization scheme efficiently.

Built on

Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.
Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients
Original
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou · 2009
Earlier work this paper cites.
Estimating or propagating gradients through stochastic neurons for conditional computation
Original
Yoshua Bengio, Nicholas Léonard, and Aaron Courville · 2013
Earlier work this paper cites.
On the importance of initialization and momentum in deep learning
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton · 2013
Earlier work this paper cites.
Speeding up convolutional neural networks with low rank expansions
Original
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman · 2014
Earlier work this paper cites.
Statistical learning with sparsity: the lasso and generalizations
Trevor Hastie, Robert Tibshirani, and Martin Wainwright · 2015
Earlier work this paper cites.
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei · 2015
Earlier work this paper cites.
Accelerating very deep convolutional networks for classification and detection
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun · 2015
Earlier work this paper cites.
Xnor-net: Imagenet classification using binary convolutional neural networks
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi · 2016
Earlier work this paper cites.

Similar

Rethinking the inception architecture for computer vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna · 2016
Cited alongside, same era.
Learning structured sparsity in deep neural networks
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li · 2016
Cited alongside, same era.
Coordinating filters for faster deep neural networks
Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li · 2017
Cited alongside, same era.
Incremental network quantization: Towards lossless cnns with low-precision weights
Original
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen · 2017
Cited alongside, same era.
Pact: Parameterized clipping activation for quantized neural networks
Original
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan · 2018
Cited alongside, same era.
Quantization and training of neural networks for efficient integer-arithmetic-only inference
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko · 2018
Cited alongside, same era.
Value-aware quantization for training and inference of neural networks
Eunhyeok Park, Sungjoo Yoo, and Peter Vajda · 2018
Cited alongside, same era.
Model compression via distillation and quantization
Original
A. Polino, R. Pascanu, and D. Alistarh · 2018
Cited alongside, same era.
An analytical method to determine minimum per-layer precision of deep neural networks
Charbel Sakr and Naresh Shanbhag · 2018
Cited alongside, same era.
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Original
Song Han, Huizi Mao, and William J Dally
Cited in the paper.
Learning both weights and connections for efficient neural network
Song Han, Jeff Pool, John Tran, and William Dally
Cited in the paper.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun
Cited in the paper.

Then

Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network
Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Vikas Chandra, and Hadi Esmaeilzadeh · 2018
Later among the works it cites.
Trained rank pruning for efficient deep neural networks
Original
Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong · 2018
Later among the works it cites.
Lq-nets: Learned quantization for highly accurate and compact deep neural networks
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua · 2018
Later among the works it cites.
Hawq: Hessian aware quantization of neural networks with mixed-precision
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer · 2019
Later among the works it cites.
Learned step size quantization
Original
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha · 2019
Later among the works it cites.
Per-tensor fixed-point quantization of the back-propagation algorithm
Charbel Sakr and Naresh Shanbhag · 2019
Later among the works it cites.
Haq: Hardware-aware automated quantization with mixed precision
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han · 2019
Later among the works it cites.
Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer · 2019
Later among the works it cites.
Deephoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures
Huanrui Yang, Wei Wen, and Hai Li · 2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…