Fetching the paper…

Reading the bibliography…

2018

Low-Precision Floating-Point Schemes for Neural Network Training

Ortiz, Marc, Cristal, Adrián, Ayguadé, Eduard et al.

Original
PDF
Post on X

Understand

The use of low-precision fixed-point arithmetic along with stochastic rounding has been proposed as a promising alternative to the commonly used 32-bit floating point arithmetic to enhance training neural networks training in terms of performance and energy efficiency.

In the first part of this paper, the behaviour of the 12-bit fixed-point arithmetic when training a convolutional neural network with the CIFAR-10 dataset is analysed, showing that such arithmetic is not the most appropriate for the training phase.
After that, the paper presents and evaluates, under the same conditions, alternative low-precision arithmetics, starting with the 12-bit floating-point arithmetic.
These two representations are then leveraged using local scaling in order to increase accuracy and get closer to the baseline 32-bit floating-point arithmetic.

Low-Precision Floating-Point Schemes for Neural Network Training · Around

Built on

Iwata, Yoshida, and Matsuda, “ An artificial neural network accelerator using general purpose 24 bit floating point digital signal processors
1989
Earlier work this paper cites.
D. Hammerstrom, “ A VLSI architecture for high-performance, low-cost, on-chip learning
1990
Earlier work this paper cites.
V. Vanhoucke, A. Senior, and M. Z. Mao, “ Improving the speed of neural networks on CPUs
2011
Earlier work this paper cites.
C. Szegedy, Y. J. Wei Liu, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “ Deep Residual Learning for Image Recognition
Original
2014
Earlier work this paper cites.
M. Courbariaux, Y. Bengio, and J.-P. David, “ Low precision arithmetic for deep learning
Original
2014
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “ Deep Residual Learning for Image Recognition
Original
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “ Deep Residual Learning for Image Recognition
Original
2015
Earlier work this paper cites.

Similar

S. Gupta, A. Agrawal, and P. N. Kailash Gopalakrishnan, “ Deep Learning with Limited Numerical Precision
Original
2015
Cited alongside, same era.
M. Courbariaux, J.-P. David, and Y. Bengio, “ Training Deep Neural Networks with Low Precision Multiplications
Original
2015
Cited alongside, same era.
F. Li, B. Zhang, and B. Liu, “ Ternary Weight Networks
Original
2016
Cited alongside, same era.
M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “ Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
Original
2016
Cited alongside, same era.
P. Judd, J. Albericio, T. Hetherington, T. Aamodt, N. E. Jerger, R. Urtasun, and A. Moshovos, “ Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets

Then

K. Guo, S. Zeng, J. Yu, Y. Wang, and H. Yang, “ A Survey of FPGA Based Neural Network Accelerator
Original
2017
Later among the works it cites.
D. D. Lin, S. S. Talathi, and V. S. Annapureddy, “ Fixed Point Quantization of Deep Convolutional Networks
Original
2017
Later among the works it cites.
X. Chen, X. Hu, N. Xu, H. Zhou, H. Zhou, and N. Xu, “ FxpNet: Training deep convolutional neural network in fixed-point representation
2017
Later among the works it cites.
U. Köster, T. J. Webb, X. Wang, M. Nassar, A. K. Bansal, W. H. Constable, O. H. Elibol, S. Gray, S. Hall, L. Hornof, A. Khosrowshahi, C. Kloss, R. J. Pai, and N. Rao, “ Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
Original
2017
Later among the works it cites.
Y. Gong, L. Liu, M. Yang, and L. Bourdev, “ Compressing Deep Convolutional Networks using Vector Quantization

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2016

Cited alongside, same era.

W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “ Achieving Human Parity in Conversational Speech Recognition

2017

Cited alongside, same era.

Original

2017

Later among the works it cites.

S. Wu, G. Li, F. Chen, and L. Shi, “ Training and Inference with Integers in Deep Neural Networks

2018

Closest in time.