Fetching the paper…

Reading the bibliography…

2020

FracBits: Mixed Precision Quantization via Fractional Bit-Widths

Yang, Linjie, Jin, Qing

Original
PDF
Post on X

Understand

Model quantization helps to reduce model size and latency of deep neural networks.

Mixed precision quantization is favorable with customized hardwares supporting arithmetic operations at multiple bit-widths to achieve maximum efficiency.
We propose a novel learning-based algorithm to derive mixed precision models end-to-end under target computation constraints and model sizes.
During the optimization, the bit-width of each layer / kernel in the model is at a fractional status of two consecutive bit-widths which can be adjusted gradually.

FracBits: Mixed Precision Quantization via Fractional Bit-Widths · Around

Built on

Network Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel Numbers
Original
Yu, J.; and Huang, T. S. 2019 · 1903
Earlier work this paper cites.
Single Path One-Shot Neural Architecture Search with Uniform Sampling
Original
Guo, Z.; Zhang, X.; Mu, H.; Heng, W.; Liu, Z.; Wei, Y.; and Sun, J. 2019 · 1904
Earlier work this paper cites.
AdaBits: Neural Network Quantization with Adaptive Bit-Widths
Original
Jin, Q.; Yang, L.; and Liao, Z. 2019a · 1912
Earlier work this paper cites.
Towards Efficient Training for Neural Network Quantization
Original
Jin, Q.; Yang, L.; and Liao, Z. 2019b · 1912
Earlier work this paper cites.
AtomNAS: Fine-Grained End-to-End Neural Architecture Search
Original
Mei, J.; Li, Y.; Lian, X.; Jin, X.; Yang, L.; Yuille, A.; and Yang, J. 2019 · 1912
Earlier work this paper cites.
BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
Original
Nikolić, M.; Hacene, G. B.; Bannon, C.; Lascorz, A. D.; Courbariaux, M.; Bengio, Y.; Gripon, V.; and Moshovos, A. 2020 · 2002
Earlier work this paper cites.
Estimating or propagating gradients through stochastic neurons for conditional computation
Original
Bengio, Y.; Léonard, N.; and Courville, A. 2013 · 2013
Earlier work this paper cites.
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Original
Han, S.; Mao, H.; and Dally, W. J. 2015 · 2015
Earlier work this paper cites.
Xnor-net: Imagenet classification using binary convolutional neural networks
Rastegari, M.; Ordonez, V.; Redmon, J.; and Farhadi, A. 2016 · 2016
Earlier work this paper cites.
Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients
Original
Zhou, S.; Wu, Y.; Ni, Z.; Zhou, X.; Wen, H.; and Zou, Y. 2016 · 2016
Earlier work this paper cites.
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Original
Goyal, P.; Dollár, P.; Girshick, R. B.; Noordhuis, P.; Wesolowski, L.; Kyrola, A.; Tulloch, A.; Jia, Y.; and He, K. 2017 · 2017
Earlier work this paper cites.

Similar

Channel pruning for accelerating very deep neural networks
He, Y.; Zhang, X.; and Sun, J. 2017 · 2017
Cited alongside, same era.
Learning efficient convolutional networks through network slimming
Liu, Z.; Li, J.; Shen, Z.; Huang, G.; Yan, S.; and Zhang, C. 2017 · 2017
Cited alongside, same era.
Thinet: A filter level pruning method for deep neural network compression
Luo, J.-H.; Wu, J.; and Lin, W. 2017 · 2017
Cited alongside, same era.
Balanced quantization: An effective and efficient approach to quantized neural networks
Original
Zhou, S.-C.; Wang, Y.-Z.; Wen, H.; He, Q.-Y.; and Zou, Y.-H. 2017 · 2017
Cited alongside, same era.
Proxylessnas: Direct neural architecture search on target task and hardware
Original
Cai, H.; Zhu, L.; and Han, S. 2018 · 2018
Cited alongside, same era.

Then

Mixed precision quantization of convnets via differentiable neural architecture search
Original
Wu, B.; Wang, Y.; Zhang, P.; Tian, Y.; Vajda, P.; and Keutzer, K. 2018 · 2018
Later among the works it cites.
SNAS: stochastic neural architecture search
Original
Xie, S.; Zheng, H.; Liu, C.; and Lin, L. 2018 · 2018
Later among the works it cites.
Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers
Original
Ye, J.; Lu, X.; Lin, Z.; and Wang, J. Z. 2018 · 2018
Later among the works it cites.
Slimmable neural networks
Original
Yu, J.; Yang, L.; Xu, N.; Yang, J.; and Huang, T. 2018 · 2018
Later among the works it cites.
Lq-nets: Learned quantization for highly accurate and compact deep neural networks

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Pact: Parameterized clipping activation for quantized neural networks

Original

Choi, J.; Wang, Z.; Venkataramani, S.; Chuang, P. I.-J.; Srinivasan, V.; and Gopalakrishnan, K. 2018 · 2018

Cited alongside, same era.

ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks

Elthakeb, A. T.; Pilligundla, P.; Yazdanbakhsh, A.; Kinzer, S.; and Esmaeilzadeh, H. 2018 · 2018

Cited alongside, same era.

Morphnet: Fast & simple resource-constrained structure learning of deep networks

Gordon, A.; Eban, E.; Nachum, O.; Chen, B.; Wu, H.; Yang, T.-J.; and Choi, E. 2018 · 2018

Cited alongside, same era.

Quantizing deep convolutional networks for efficient inference: A whitepaper

Krishnamoorthi, R. 2018 · 2018

Cited alongside, same era.

Efficient neural architecture search via parameter sharing

Pham, H.; Guan, M. Y.; Zoph, B.; Le, Q. V.; and Dean, J. 2018 · 2018

Cited alongside, same era.

Zhang, D.; Yang, J.; Ye, D.; and Hua, G. 2018 · 2018

Later among the works it cites.

Data-free quantization through weight equalization and bias correction

Nagel, M.; Baalen, M. v.; Blankevoort, T.; and Welling, M. 2019 · 2019

Later among the works it cites.

Haq: Hardware-aware automated quantization with mixed precision

Wang, K.; Liu, Z.; Lin, Y.; Lin, J.; and Han, S. 2019 · 2019

Later among the works it cites.

Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search

Wu, B.; Dai, X.; Zhang, P.; Wang, Y.; Sun, F.; Wu, Y.; Tian, Y.; Vajda, P.; Jia, Y.; and Keutzer, K. 2019 · 2019

Later among the works it cites.

AutoQ: Automated Kernel-Wise Neural Network Quantization

Lou, Q.; Guo, F.; Kim, M.; Liu, L.; and Jiang., L. 2020 · 2020

Closest in time.

MIXED PRECISION DNNS: ALL YOU NEED IS A GOOD PARAMETRIZATION

Uhlich, S.; Mauch, L.; Yoshiyama, K.; Cardinaux, F.; Garcia, J. A.; Tiedemann, S.; Kemp, T.; and Nakamura, A. 2020 · 2020

Closest in time.