Fetching the paper…

Reading the bibliography…

2018

Implementation of Stochastic Quasi-Newton's Method in PyTorch

Li, Yingkai, Liu, Huidong

Original
PDF
Post on X

Understand

In this paper, we implement the Stochastic Damped LBFGS (SdLBFGS) for stochastic non-convex optimization.

We make two important modifications to the original SdLBFGS algorithm.
First, by initializing the Hessian at each step using an identity matrix, the algorithm converges better than original algorithm.
Second, by performing direction normalization we could gain stable optimization procedure without line search.

Implementation of Stochastic Quasi-Newton's Method in PyTorch · Around

Built on

On the limited memory bfgs method for large scale optimization
D. C. Liu and J. Nocedal · 1989
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner · 1998
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky and G. Hinton · 2009
Earlier work this paper cites.

Similar

Large-scale machine learning with stochastic gradient descent
L. Bottou · 2010
Cited alongside, same era.
Adaptive subgradient methods for online learning and stochastic optimization
J. Duchi, E. Hazan, and Y. Singer · 2011
Cited alongside, same era.
Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude
T. Tieleman and G. Hinton · 2012
Cited alongside, same era.

Then

Adadelta: an adaptive learning rate method
Original
M. D. Zeiler · 2012
Later among the works it cites.
Adam: A method for stochastic optimization
Original
D. P. Kingma and J. Ba · 2014
Later among the works it cites.
Stochastic quasi-newton methods for nonconvex stochastic optimization
X. Wang, S. Ma, D. Goldfarb, and W. Liu · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…