2017

Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization

Dong, Yinpeng, Ni, Renkun, Li, Jianguo et al.

Understand

Low-bit deep neural networks (DNNs) become critical for embedded applications due to their low storage requirement and computing efficiency.

However, they suffer much from the non-negligible accuracy drop.
This paper proposes the stochastic quantization (SQ) algorithm for learning accurate low-bit DNNs.
The motivation is due to the following observation.

Built on

Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Predicting parameters in deep learning
Misha Denil, Babak Shakibi, Laurent Dinh, Nando de Freitas, et al · 2013
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik · 2014
Earlier work this paper cites.
Caffe: Convolutional architecture for fast feature embedding
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell · 2014
Earlier work this paper cites.
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Earlier work this paper cites.
Semantic image segmentation with deep convolutional nets and fully connected crfs
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille · 2015
Earlier work this paper cites.
Binaryconnect: Training deep neural networks with binary weights during propagations
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David · 2015
Earlier work this paper cites.
Learning both weights and connections for efficient neural network
Song Han, Jeff Pool, John Tran, and William Dally · 2015
Earlier work this paper cites.

Similar

Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton · 2015
Cited alongside, same era.
Fully convolutional networks for semantic segmentation
Jonathan Long, Evan Shelhamer, and Trevor Darrell · 2015
Cited alongside, same era.
Faster r-cnn: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al · 2015
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman · 2015
Cited alongside, same era.
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich · 2015
Cited alongside, same era.
Binarynet: Training deep neural networks with weights and activations constrained to+ 1 or-1
Original
Matthieu Courbariaux and Yoshua Bengio · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Deep networks with stochastic depth
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger · 2016
Cited alongside, same era.

Then

Quantized neural networks: Training neural networks with low precision weights and activations
Original
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio · 2016
Later among the works it cites.
Ternary weight networks
Fengfu Li, Bo Zhang, and Bin Liu · 2016
Later among the works it cites.
Fixed point quantization of deep convolutional networks
Darryl D Lin, Sachin S Talathi, and V Sreekanth Annapureddy · 2016
Later among the works it cites.
Convolutional neural networks using logarithmic data representation
Original
Daisuke Miyashita, Edward H Lee, and Boris Murmann · 2016
Later among the works it cites.
Xnor-net: Imagenet classification using binary convolutional neural networks
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi · 2016
Later among the works it cites.
Accelerating deep convolutional networks using low-precision and sparsity
Original
Ganesh Venkatesh, Eriko Nurvitadhi, and Debbie Marr · 2016
Later among the works it cites.
Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients
Original
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou · 2016
Later among the works it cites.
Incremental network quantization: Towards lossless cnns with low-precision weights
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen · 2017
Closest in time.
Trained ternary quantization
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…