2018

Fast Adjustable Threshold For Uniform Neural Network Quantization (Winning solution of LPIRC-II)

Goncharenko, Alexander, Denisov, Andrey, Alyamkin, Sergey et al.

Understand

Neural network quantization procedure is the necessary step for porting of neural networks to mobile devices.

Quantization allows accelerating the inference, reducing memory consumption and model size.
It can be performed without fine-tuning using calibration procedure (calculation of parameters necessary for quantization), or it is possible to train the network with quantization from scratch.
Training with quantization from scratch on the labeled data is rather long and resource-consuming procedure.

Built on

Bengio, Y., Leonard, N., and Courville, A. C. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013)
Original
2013
Earlier work this paper cites.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. Imagenet large scale visual recognition challenge. arXiv preprint arXiv:1409.0575 (2014)
Original
2014
Earlier work this paper cites.
Courbariaux, M., Bengio, Y., and David, J. Training deep neural networks with low precision multiplications. In: International Conference on Learning Representations, ICLR (2015)
2015
Earlier work this paper cites.
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Original
2015
Earlier work this paper cites.
Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, ICML (2015)
2015
Earlier work this paper cites.
Kingma, D. P. and Ba, J. L. Adam: A method for stochastic optimization. In: International Conference on Learning Representations, ICLR (2015)
2015
Earlier work this paper cites.
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mane, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viegas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. Tensorflow: Largescale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016)
Original
2016
Earlier work this paper cites.
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Binarized neural networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 4107–4115 (2016)
2016
Earlier work this paper cites.

Similar

Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. Xnor-net: Imagenet classification using binary convolutional neural networks. In: European Conference on Computer Vision (ECCV), pp. 525–542. Springer (2016)
2016
Cited alongside, same era.
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 (2016)
Original
2016
Cited alongside, same era.
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D.,Wang, W.,Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Original
2017
Cited alongside, same era.
Mishra, A. and Marr, D. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. arXiv preprint arXiv:1711.05852 (2017)
Original
2017
Cited alongside, same era.
Mishra, A., Nurvitadhi, E., Cook, J. J., and Marr, D. Wrpn: Wide reduced-precision networks. arXiv preprint arXiv:1709.01134 (2017)
Original
2017
Cited alongside, same era.
https://developer.nvidia.com/tensorrt (2018)
2018
Cited alongside, same era.
Baskin, C., Liss, N., Chai, Y., Zheltonozhskii, E., Schwartz, E., Giryes, R., Mendelson, A., and Bronstein, A. M. Nice: Noise injection and clamping estimation for neural network quantization. arXiv preprint arXiv:1810.00162 (2018)
Original
2018
Cited alongside, same era.
https://github.com/nervanasystems/distiller
Cited in the paper.
https://github.com/tensorflow/tensorflow/blob/ 61c6c84964b4aec80aeace187aab8cb2c3e55a72/ tensorflow/lite/g3doc/models.md
Cited in the paper.

Then

Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer-arithmetic only inference. In: Conference on Computer Vision and Pattern Recognition, CVPR (2018)
2018
Closest in time.
Lee, J. H., Ha, S., Choi, S., Lee, W., and Lee, S. Quantization for rapid deployment of deep neural networks. arXiv preprint arXiv:1810.05488 (2018)
Original
2018
Closest in time.
McDonnell, M. D. Training wide residual networks for deployment using a single bit for each weight. In: International Conference on Learning Representations, ICLR (2018)
2018
Closest in time.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2018)
2018
Closest in time.
Sheng, T., Feng, C., Zhuo, S., Zhang, X., Shen, L., and Aleksic, M. A quantization-friendly separable convolution for mobilenets. arXiv preprint arXiv:1803.08607 (2018)
Original
2018
Closest in time.
Tan, M., Chen, B., Pang, R., Vasudevan, V., and Le, Q. V. Mnasnet: Platform-aware neural architecture search for mobile. arXiv preprint arXiv:1807.11626 (2018)
Original
2018
Closest in time.
Zhu, S., Dong, X., and Su, H. Binary ensemble neural network: More bits per network or more networks per bit? arXiv preprint arXiv:1806.07550 (2018)
Original
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…