Understand
Humans are able to accelerate their learning by selecting training materials that are the most informative and at the appropriate level of difficulty.
- We propose a framework for distributing deep learning in which one set of workers search for the most informative examples in parallel while a single worker updates the model on examples selected by importance sampling.
- This leads the model to update using an unbiased estimate of the gradient which also has minimum variance when the sampling proposal is proportional to the L2-norm of the gradient.
- We show experimentally that this method reduces gradient variance even in a context where the cost of synchronization across machines cannot be ignored, and where the factors for importance sampling are not updated instantly across the training set.
Built on
Importance sampling: a review
Tokdar, Surya T. and Kass, Robert E · 1939
Earlier work this paper cites.
A neural probabilistic language model
Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C · 2003
Earlier work this paper cites.
Curriculum learning
Bengio, Yoshua, Louradour, Jerome, Collobert, Ronan, and Weston, Jason · 2009
Earlier work this paper cites.
Theano: a CPU and GPU math expression compiler
Bergstra, James, Breuleux, Olivier, Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua · 2010
Earlier work this paper cites.
Distributed delayed stochastic optimization
Agarwal, Alekh and Duchi, John C · 2011
Earlier work this paper cites.
Reading digits in natural images with unsupervised feature learning
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y · 2011
Earlier work this paper cites.
Similar
Hogwild: A lock-free approach to parallelizing stochastic gradient descent
Recht, Benjamin, Re, Christopher, Wright, Stephen, and Niu, Feng · 2011
Cited alongside, same era.
Theano: new features and speed improvements
Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian J., Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua · 2012
Cited alongside, same era.
Large scale distributed deep networks
Dean, J., Corrado, G.S, Monga, R., Chen, K., Devin, M., Le, Q.V., Mao, M.Z., Ranzato, M.A., Senior, A., Tucker, P., Yang, K., and Ng, A. Y · 2012
Cited alongside, same era.
Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks
Davis, A. and Arel, I · 2013
Cited alongside, same era.
Goodfellow, Ian J., Warde-Farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua · 2013
Cited alongside, same era.
Then
Compete to compute
Srivastava, Rupesh K, Masci, Jonathan, Kazerounian, Sohrob, Gomez, Faustino, and Schmidhuber, Jürgen · 2013
Later among the works it cites.
Stochastic Optimization with Importance Sampling
Zhao, P. and Zhang, T · 2014
Later among the works it cites.
Accelerating stochastic gradient descent via online learning to sample
Bouchard, Guillaume, Trouillon, Théo, Perez, Julien, and Gaidon, Adrien · 2015
Closest in time.
Efficient Per-Example Gradient Computations
Goodfellow, I · 2015
Closest in time.
Asynchronous parallel stochastic gradient for nonconvex optimization
Lian, Xiangru, Huang, Yijun, Li, Yuncheng, and Liu, Ji · 2015
Closest in time.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…