Asynchronous Distributed Semi-Stochastic Gradient Optimization

Understand

With the recent proliferation of large-scale learning problems,there have been a lot of interest on distributed machine learning algorithms, particularly those that are based on stochastic gradient descent (SGD) and its variants.

• However, existing algorithms either suffer from slow convergence due to the inherent variance of stochastic gradients, or have a fast linear convergence rate but at the expense of poorer solution quality.
• In this paper, we combine their merits by proposing a fast distributed asynchronous SGD-based algorithm with variance reduction.
• A constant learning rate can be used, and it is also guaranteed to converge linearly to the optimal solution.