Built on

S.-I. Amari, “Natural gradient works efficiently in learning,”
1998
Earlier work this paper cites.
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” tech. rep., Citeseer, 2009
2009
Earlier work this paper cites.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011
2011
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in
2012
Earlier work this paper cites.
J. Martens and R. B. Grosse, “Optimizing neural networks with Kronecker-factored Approximate Curvature,”
Original
2015
Earlier work this paper cites.
N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,”
Original
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in
2016
Earlier work this paper cites.
R. Grosse and J. Martens, “A Kronecker-factored Approximate Fisher matrix for convolution layers,” in
2016
Earlier work this paper cites.
Preprint: arXiv:1601.04737
Original
F. Roosta-Khorasani and M. W. Mahoney, “Sub-sampled Newton methods I: Globally convergent algorithms,” tech. rep., 2016 · 2016
Earlier work this paper cites.
Preprint: arXiv:1601.04738
Original
F. Roosta-Khorasani and M. W. Mahoney, “Sub-sampled Newton methods II: Local convergence rates,” tech. rep., 2016 · 2016
Earlier work this paper cites.

Similar

S. Ma, R. Bassily, and M. Belkin, “The power of interpolation: Understanding the effectiveness of SGD in modern over-parametrized learning,”
Original
2017
Cited alongside, same era.
P. Goyal, P. Dollár, R. B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch SGD: Training imagenet in 1 hour,”
Original
2017
Cited alongside, same era.
S. L. Smith, P.-J. Kindermans, C. Ying, and Q. V. Le, “Don’t decay the learning rate, increase the batch size,”
Original
2017
Cited alongside, same era.
Y. You, Z. Zhang, C. Hsieh, and J. Demmel, “100-epoch imagenet training with alexnet in 24 minutes,”
Original
2017
Cited alongside, same era.
J. Ba, R. Grosse, and J. Martens, “Distributed second-order optimization using Kronecker-factored Approximations,” in

Then

S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team, “An empirical model of large-batch training,”
Original
2018
Later among the works it cites.
C. J. Shallue, J. Lee, J. M. Antognini, J. Sohl-Dickstein, R. Frostig, and G. E. Dahl, “Measuring the effects of data parallelism on neural network training,”
Original
2018
Later among the works it cites.
Z. Yao, A. Gholami, K. Keutzer, and M. W. Mahoney, “Large batch size training of neural networks with adversarial training and second-order information,”
Original
2018
Later among the works it cites.
N. Mu, Z. Yao, A. Gholami, K. Keutzer, and M. W. Mahoney, “Parameter re-initialization through cyclical batch size schedules,”
Original
2018
Later among the works it cites.
K. Osawa, Y. Tsuji, Y. Ueno, A. Naruse, R. Yokota, and S. Matsuoka, “Second-order optimization method for large mini-batch: Training resnet-50 on imagenet in 35 epochs,”

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2019

Inefficiency of K-FAC for Large Batch Size Training

Ma, Linjian, Montague, Gabe, Ye, Jiayu et al.

Understand

In stochastic optimization, using large batch sizes during training can leverage parallel resources to produce faster wall-clock training times per training epoch.

However, for both training loss and testing error, recent results analyzing large batch Stochastic Gradient Descent (SGD) have found sharp diminishing returns, beyond a certain critical batch size.
In the hopes of addressing this, it has been suggested that the Kronecker-Factored Approximate Curvature (\mbox{K-FAC}) method allows for greater scalability to large batch sizes, for non-convex machine learning problems such as neural network optimization, as well as greater robustness to variation in model hyperparameters.
Here, we perform a detailed empirical analysis of large batch size training %of these two hypotheses, for both \mbox{K-FAC} and SGD, evaluating performance in terms of both wall-clock time and aggregate computational cost.

Built on

S.-I. Amari, “Natural gradient works efficiently in learning,”
1998
Earlier work this paper cites.
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” tech. rep., Citeseer, 2009
2009
Earlier work this paper cites.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011
2011
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in
2012
Earlier work this paper cites.
J. Martens and R. B. Grosse, “Optimizing neural networks with Kronecker-factored Approximate Curvature,”
Original
2015
Earlier work this paper cites.
N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,”
Original
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in
2016
Earlier work this paper cites.
R. Grosse and J. Martens, “A Kronecker-factored Approximate Fisher matrix for convolution layers,” in
2016
Earlier work this paper cites.
Preprint: arXiv:1601.04737
Original
F. Roosta-Khorasani and M. W. Mahoney, “Sub-sampled Newton methods I: Globally convergent algorithms,” tech. rep., 2016 · 2016
Earlier work this paper cites.
Preprint: arXiv:1601.04738
Original
F. Roosta-Khorasani and M. W. Mahoney, “Sub-sampled Newton methods II: Local convergence rates,” tech. rep., 2016 · 2016
Earlier work this paper cites.