2018

High-Accuracy Low-Precision Training

De Sa, Christopher, Leszczynski, Megan, Zhang, Jian et al.

Understand

Low-precision computation is often used to lower the time and energy cost of machine learning, and recently hardware accelerators have been developed to support it.

Still, it has been used primarily for inference - not training.
Previous low-precision training algorithms suffered from a fundamental tradeoff: as the number of bits of precision is lowered, quantization noise is added to the model, which limits statistical accuracy.
To address this issue, we describe a simple low-precision stochastic gradient descent variant called HALP.

Built on

Learning representations by back-propagating errors
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams · 1986
Earlier work this paper cites.
Stochastic gradient learning in neural networks
Léon Bottou · 1991
Earlier work this paper cites.
Xorshift RNGs
George Marsaglia · 2003
Earlier work this paper cites.
Scikit-learn: Machine learning in Python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay · 2011
Earlier work this paper cites.
Resource efficient arithmetic effects on RBM neural network solution quality using MNIST
Antony W Savich and Medhat Moussa · 2011
Earlier work this paper cites.
Improving the speed of neural networks on CPUs
Vincent Vanhoucke, Andrew Senior, and Mark Z. Mao · 2011
Earlier work this paper cites.
Stochastic gradient descent tricks
Léon Bottou · 2012
Earlier work this paper cites.
A stochastic gradient method with an exponential convergence rate for finite training sets
Nicolas L Roux, Mark Schmidt, and Francis R Bach · 2012
Earlier work this paper cites.
Accelerating stochastic gradient descent using predictive variance reduction
Rie Johnson and Tong Zhang · 2013
Earlier work this paper cites.
Stochastic dual coordinate ascent methods for regularized loss minimization
Shai Shalev-Shwartz and Tong Zhang · 2013
Earlier work this paper cites.
Training deep neural networks with low precision multiplications
Original
Matthieu Courbariaux, Jean-Pierre David, and Yoshua Bengio · 2014
Earlier work this paper cites.

Similar

The CIFAR-10 dataset
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton · 2014
Cited alongside, same era.
1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs
Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu · 2014
Cited alongside, same era.
Fixed point optimization of deep convolutional neural networks for object recognition
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung · 2015
Cited alongside, same era.
Taming the wild: A unified analysis of Hogwild!
Christopher De Sa, Ce Zhang, Kunle Olukotun, and Christopher Ré · 2015
Cited alongside, same era.
Deep learning with limited numerical precision
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan · 2015
Cited alongside, same era.
Stop wasting my gradients: Practical SVRG
Reza Harikandeh, Mohamed Osama Ahmed, Alim Virani, Mark Schmidt, Jakub Konečnỳ, and Scott Sallinen · 2015
Cited alongside, same era.
Visualizing and understanding recurrent networks
Original
Andrej Karpathy, Justin Johnson, and Li Fei-Fei · 2015
Cited alongside, same era.
Scalable distributed DNN training using commodity GPU cloud computing
Nikko Strom · 2015
Cited alongside, same era.
Variance reduction for faster non-convex optimization
Zeyuan Allen-Zhu and Elad Hazan · 2016
Cited alongside, same era.
Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding
Song Han, Huizi Mao, and William J Dally · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.

Then

Stochastic variance reduction for nonconvex optimization
Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola · 2016
Later among the works it cites.
QSGD: Communication-efficient SGD via gradient quantization and encoding
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic · 2017
Later among the works it cites.
Configurable clouds
Adrian M Caulfield, Eric S Chung, Andrew Putnam, Hari Angepat, Daniel Firestone, Jeremy Fowers, Michael Haselman, Stephen Heil, Matt Humphrey, Puneet Kaur, et al · 2017
Later among the works it cites.
Understanding and optimizing asynchronous low-precision stochastic gradient descent
Christopher De Sa, Matthew Feldman, Christopher Ré, and Kunle Olukotun · 2017
Later among the works it cites.
In-datacenter performance analysis of a tensor processing unit
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al · 2017
Later among the works it cites.
TensorQuant: A simulation toolbox for deep neural network quantization
Dominik Marek Loroch, Franz-Josef Pfreundt, Norbert Wehn, and Janis Keuper · 2017
Later among the works it cites.
Mixed precision training
Original
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu · 2017
Later among the works it cites.
The ZipML framework for training models with end-to-end low precision: The cans, the cannots, and a little bit of deep learning
H Zhang, J Li, K Kara, D Alistarh, J Liu, and C Zhang · 2017
Later among the works it cites.
Microsoft unveils Project Brainwave for real-time ai
Doug Burger · 2018
Closest in time.
Transparent model distillation
Original
Sarah Tan, Rich Caruana, Giles Hooker, and Albert Gordo · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…