A Multi-Batch L-BFGS Method for Machine Learning

Understand

The question of how to parallelize the stochastic gradient descent (SGD) method has received much attention in the literature.

• In this paper, we focus instead on batch methods that use a sizeable fraction of the training set at each iteration to facilitate parallelism, and that employ second-order information.
• In order to improve the learning process, we follow a multi-batch approach in which the batch changes at each iteration.
• This can cause difficulties because L-BFGS employs gradient differences to update the Hessian approximations, and when these gradients are computed using different data points the process can be unstable.