Fetching the paper…
Reading the bibliography…
Many popular distributed optimization methods for training machine learning models fit the following template: a local gradient estimate is computed independently by each worker, then communicated to a master, which subsequently performs averaging.
A method for solving the convex programming problem with convergence rate O(1/kˆ2)
Yurii Nesterov · 1983
Earlier work this paper cites.
A stochastic approximation method
Herbert Robbins and Sutton Monro · 1985
Earlier work this paper cites.
Introductory lectures on convex optimization: A basic course
Yurii Nesterov · 2004
Earlier work this paper cites.
A first-order primal-dual algorithm for convex problems with applications to imaging
Antonin Chambolle and Thomas Pock · 2011
Earlier work this paper cites.
LibSVM: A library for support vector machines
Chih-Chung Chang and Chih-Jen Lin · 2011
Earlier work this paper cites.
Parallel distributed computing using Python
Lisandro D Dalcin, Rodrigo R Paz, Pablo A Kler, and Alejandro Cosimo · 2011
Earlier work this paper cites.
Hogwild: A lock-free approach to parallelizing stochastic gradient descent
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu · 2011
Earlier work this paper cites.
Efficiency of coordinate descent methods on huge-scale optimization problems
Yu Nesterov · 2012
Earlier work this paper cites.
Accelerating stochastic gradient descent using predictive variance reduction
Rie Johnson and Tong Zhang · 2013
Earlier work this paper cites.
SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien · 2014
Earlier work this paper cites.
Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function
Peter Richtárik and Martin Takáč · 2014
Earlier work this paper cites.
Communication-efficient distributed optimization using an approximate Newton-type method
Ohad Shamir, Nati Srebro, and Tong Zhang · 2014
Earlier work this paper cites.
Accelerated mini-batch randomized block coordinate descent method
Tuo Zhao, Mo Yu, Yiming Wang, Raman Arora, and Han Liu · 2014
Earlier work this paper cites.
Accelerated, parallel, and proximal coordinate descent
Olivier Fercoq and Peter Richtárik · 2015
Earlier work this paper cites.
Randomized iterative methods for linear systems
Robert M Gower and Peter Richtárik · 2015
Cited alongside, same era.
Adding vs. averaging in distributed primal-dual optimization
Chenxin Ma, Virginia Smith, Martin Jaggi, Michael I. Jordan, Peter Richtárik, and Martin Takáč · 2015
Cited alongside, same era.
Quartz: Randomized dual coordinate ascent with arbitrary sampling
Zheng Qu, Peter Richtárik, and Tong Zhang · 2015
Cited alongside, same era.
Coordinate descent algorithms
Stephen J Wright · 2015
Cited alongside, same era.
A simple practical accelerated method for finite sums
Aaron Defazio · 2016
Cited alongside, same era.
Coordinate descent with arbitrary sampling I: Algorithms and complexity
Zheng Qu and Peter Richtárik · 2016
Cited alongside, same era.
SignSGD: Compressed optimisation for non-convex problems
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar · 2018
Later among the works it cites.
Importance sampling for minibatches
Dominik Csiba and Peter Richtárik · 2018
Later among the works it cites.
Stochastic quasi-gradient methods: Variance reduction via Jacobian sketching
Robert M Gower, Peter Richtárik, and Francis Bach · 2018
Later among the works it cites.
Asynchronous distributed learning with sparse communications and identification
Dmitry Grishchenko, Franck Iutzeler, Jérôme Malick, and Massih-Reza Amini · 2018
Later among the works it cites.
SEGA: Variance reduction via gradient sketching
Filip Hanzely, Konstantin Mishchenko, and Peter Richtárik · 2018
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Coordinate descent with arbitrary sampling II: Expected separable overapproximation
Zheng Qu and Peter Richtárik · 2016
Cited alongside, same era.
Parallel coordinate descent methods for big data optimization
Peter Richtárik and Martin Takáč · 2016
Cited alongside, same era.
QSGD: Communication-efficient SGD via gradient quantization and encoding
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic · 2017
Cited alongside, same era.
Katyusha: The first direct acceleration of stochastic gradient methods
Zeyuan Allen-Zhu · 2017
Cited alongside, same era.
Asaga: Asynchronous parallel saga
Rémi Leblond, Fabian Pedregosa, and Simon Lacoste-Julien · 2017
Cited alongside, same era.
Deep gradient compression: Reducing the communication bandwidth for distributed training
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally · 2017
Cited alongside, same era.
Accelerated coordinate descent with arbitrary sampling and best rates for minibatches
Filip Hanzely and Peter Richtárik · 2018
Later among the works it cites.
Nonconvex variance reduced optimization with arbitrary sampling
Samuel Horváth and Peter Richtárik · 2018
Later among the works it cites.
Improved asynchronous parallel optimization analysis for stochastic incremental methods
Rémi Leblond, Fabian Pedregosa, and Simon Lacoste-Julien · 2018
Later among the works it cites.
A distributed flexible delay-tolerant proximal gradient algorithm
Konstantin Mishchenko, Franck Iutzeler, and Jérôme Malick · 2018
Later among the works it cites.
A delay-tolerant proximal-gradient algorithm for distributed learning
Konstantin Mishchenko, Franck Iutzeler, Jérôme Malick, and Massih-Reza Amini · 2018
Later among the works it cites.
Fast and faster convergence of SGD for over-parameterized models and an accelerated perceptron
Sharan Vaswani, Francis Bach, and Mark Schmidt · 2018
Later among the works it cites.
ATOMO: Communication-efficient learning via atomic sparsification
Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris Papailiopoulos, and Stephen Wright · 2018
Later among the works it cites.
Gradient sparsification for communication-efficient distributed optimization
Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang · 2018
Later among the works it cites.
Distributed learning with compressed gradient differences
Konstantin Mishchenko, Eduard Gorbunov, Martin Takáč, and Peter Richtárik · 2019
Closest in time.