2017

Large Batch Training of Convolutional Networks

You, Yang, Gitman, Igor, Ginsburg, Boris

Understand

A common way to speed up training of large convolutional networks is to add computational units.

Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units.
With an increase in the number of nodes, the batch size grows.
But training with large batch size often results in the lower model accuracy.

Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Lecture 6.5-rmsprop, coursera: Neural networks for machine learning
Tijmen Tieleman and Geoffrey Hinton · 2012
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
One weird trick for parallelizing convolutional neural networks
Original
Alex Krizhevsky · 2014
Earlier work this paper cites.
Efficient mini-batch training for stochastic optimization
Mu Li, Tong Zhang, Yuqiang Chen, and Alexander J Smola · 2014
Earlier work this paper cites.

Revisiting distributed synchronous sgd
Original
Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal Jozefowicz · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
On large-batch training for deep learning: Generalization gap and sharp minima
Original
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang · 2016
Cited alongside, same era.
Powerai ddl
Original
Minsik Cho, Ulrich Finkler, Sameer Kumar, David Kung, Vaibhav Saxena, and Dheeraj Sreedhar · 2017
Cited alongside, same era.
Blog: Achieving deep learning training in less than 40 minutes on imagenet-1k with scale-out intel® xeon™/xeon phi™ architectures
Valeriu Codreanu, Damian Podareanu, and Vikram Saletore · 2017
Cited alongside, same era.

Accurate, large minibatch sgd: Training imagenet in 1 hour
Original
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He · 2017
Closest in time.
Train longer, generalize better: closing the generalization gap in large batch training of neural networks
Original
Elad Hoffer, Itay Hubara, and Daniel Soudry · 2017
Closest in time.
Diagonal rescaling for neural networks
Original
Jean Lafond, Nicolas Vasilache, and Léon Bottou · 2017
Closest in time.
Scaling Distributed Machine Learning with System and Algorithm Co-design
Mu Li · 2017
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…