A Robust Adaptive Stochastic Gradient Method for Deep Learning

H. Robbins and S. Monro, “A stochastic approximation method,” The annals of mathematical statistics , pp. 400–407, 1951

1951

Earlier work this paper cites.

S. Becker and Y. Le Cun, “Improving the convergence of back-propagation learning with second order methods,” in Proceedings of the 1988 connectionist models summer school . San Matteo, CA: Morgan Kaufmann, 1988, pp. 29–37

1988

Earlier work this paper cites.

Y. LeCun, P. Y. Simard, and B. Pearlmutter, “Automatic learning rate maximization by on-line estimation of the hessian’s eigenvectors,” Advances in neural information processing systems , vol. 5, pp. 156–163, 1993

1993

Earlier work this paper cites.

Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE transactions on neural networks , vol. 5, no. 2, pp. 157–166, 1994

1994

Earlier work this paper cites.

S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput. , vol. 9, no. 8, pp. 1735–1780, Nov. 1997. [Online]. Available: http://dx.doi.org/10.1162/neco.1997.9.8.1735

1997

Earlier work this paper cites.

S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber, “Gradient flow in recurrent nets: the difficulty of learning long-term dependencies,” 2001

2001

Earlier work this paper cites.

Y. Levin and A. Ben-Israel, “Directional newton methods in n variables,” Mathematics of Computation , vol. 71, no. 237, pp. 251–262, 2002

2002

Earlier work this paper cites.

N. N. Schraudolph, “Fast curvature matrix-vector products for second-order gradient descent,” Neural computation , vol. 14, no. 7, pp. 1723–1738, 2002

2002

Earlier work this paper cites.

H.-B. An and Z.-Z. Bai, “Directional secant method for nonlinear equations,” Journal of computational and applied mathematics , vol. 175, no. 2, pp. 291–304, 2005

2005

Earlier work this paper cites.

M. Liwicki and H. Bunke, “Iam-ondb - an on-line english sentence database acquired from handwritten text on a whiteboard.” in ICDAR . IEEE Computer Society, 2005, pp. 956–961

2005

Cited alongside, same era.

J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning and stochastic optimization,” The Journal of Machine Learning Research , vol. 12, pp. 2121–2159, 2011

2011

Cited alongside, same era.

T. Schaul, S. Zhang, and Y. LeCun, “No more pesky learning rates,” arXiv preprint arXiv:1206.1106 , 2012

Original

2012

Cited alongside, same era.

Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, “Efficient backprop,” in Neural networks: Tricks of the trade . Springer, 2012, pp. 9–48

2012

Cited alongside, same era.

M. D. Zeiler, “Adadelta: An adaptive learning rate method,” arXiv preprint arXiv:1212.5701 , 2012

R. Johnson and T. Zhang, “Accelerating stochastic gradient descent using predictive variance reduction,” in Advances in Neural Information Processing Systems , 2013, pp. 315–323

2013

Later among the works it cites.

I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, “Maxout networks,” arXiv preprint arXiv:1302.4389 , 2013

Original

2013

Later among the works it cites.

A. Graves, “Generating sequences with recurrent neural networks,” arXiv preprint arXiv:1308.0850 , 2013

Original

2013

Later among the works it cites.

I. J. Goodfellow, D. Warde-Farley, P. Lamblin, V. Dumoulin, M. Mirza, R. Pascanu, J. Bergstra, F. Bastien, and Y. Bengio, “Pylearn2: a machine learning research library,” arXiv preprint arXiv:1308.4214 , 2013

Original

2013

Later among the works it cites.

C. Gulcehre, M. Moczulski, and Y. Bengio, “Adasecant: robust adaptive secant method for stochastic gradient,” arXiv preprint arXiv:1412.7419 , 2014