Understand
Asynchronous distributed stochastic gradient descent methods have trouble converging because of stale gradients.
- A gradient update sent to a parameter server by a client is stale if the parameters used to calculate that gradient have since been updated on the server.
- Approaches have been proposed to circumvent this problem that quantify staleness in terms of the number of elapsed updates.
- In this work, we propose a novel method that quantifies staleness in terms of moving averages of gradient statistics.
Built on
Gradient-based learning applied to document recognition
LeCun, Yann, Bottou, Léon, Bengio, Yoshua, and Haffner, Patrick · 1998
Earlier work this paper cites.
A neural probabilistic language model
Bengio, Yoshua, Ducharme, Réjean, Vincent, Pascal, and Jauvin, Christian · 2003
Earlier work this paper cites.
Theano: a CPU and GPU math expression compiler
Bergstra, James, Breuleux, Olivier, Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua · 2010
Earlier work this paper cites.
Large scale distributed deep networks
Dean, J., Corrado, G.S, Monga, R., Chen, K., Devin, M., Le, Q.V., Mao, M.Z., Ranzato, M.A., Senior, A., Tucker, P., Yang, K., and Ng, A. Y · 2012
Earlier work this paper cites.
Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude
Tieleman, Tijmen and Hinton, Geoffrey · 2012
Earlier work this paper cites.
Similar
Generating sequences with recurrent neural networks
Graves, Alex · 2013
Cited alongside, same era.
Distributed asynchronous optimization of convolutional neural networks
Chan, William and Lane, Ian · 2014
Cited alongside, same era.
Project adam: Building an efficient and scalable deep learning training system
Chilimbi, Trishul, Suzue, Yutaka, Apacible, Johnson, and Kalyanaraman, Karthik · 2014
Cited alongside, same era.
1-bit stochastic gradient descent and application to data-parallel distributed training of speech dnns
Seide, Frank, Fu, Hao, Droppo, Jasha, Li, Gang, and Yu, Dong · 2014
Cited alongside, same era.
Conditional computation in neural networks for faster models
Bengio, Emmanuel, Bacon, Pierre-Luc, Pineau, Joelle, and Precup, Doina · 2015
Cited alongside, same era.
Then
Model accuracy and runtime tradeoff in distributed deep learning
Gupta, S., Zhang, W., and Milthorpe, J · 2015
Later among the works it cites.
Fast algorithms for convolutional neural networks
Lavin, A. and Gray, S · 2015
Later among the works it cites.
Asynchronous parallel stochastic gradient for nonconvex optimization
Lian, Xiangru, Huang, Yijun, Li, Yuncheng, and Liu, Ji · 2015
Later among the works it cites.
Deep image: Scaling up image recognition
Wu, Ren, Yan, Shengen, Shan, Yi, Dang, Qingqing, and Sun, Gang · 2015
Later among the works it cites.
Staleness-aware async-sgd for distributed deep learning
Zhang, Wei, Gupta, Suyog, Lian, Xiangru, and Liu, Ji · 2015
Later among the works it cites.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…