2017

Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train

Codreanu, Valeriu, Podareanu, Damian, Saletore, Vikram

Understand

For the past 5 years, the ILSVRC competition and the ImageNet dataset have attracted a lot of interest from the Computer Vision community, allowing for state-of-the-art accuracy to grow tremendously.

This should be credited to the use of deep artificial neural network designs.
As these became more complex, the storage, bandwidth, and compute requirements increased.
This means that with a non-distributed approach, even when using the most high-density server available, the training process may take weeks, making it prohibitive.

Built on

Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Senior, A., Tucker, P., Yang, K., Le, Q.V. and Ng, A.Y., 2012. Large scale distributed deep networks. In Advances in neural information processing systems
2012
Earlier work this paper cites.
Krizhevsky, A., Sutskever, I. and Hinton, G.E., 2012. ImageNet classification with deep convolutional neural networks. In Advances in neural information processing systems
2012
Earlier work this paper cites.
Chilimbi, T.M., Suzue, Y., Apacible, J. and Kalyanaraman, K., 2014, October. Project Adam: Building an Efficient and Scalable Deep Learning Training System. In OSDI
2014
Earlier work this paper cites.
Ioffe, S. and Szegedy, C., 2015, June. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning
2015
Earlier work this paper cites.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. and Berg, A.C., 2015. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), pp.211-252
2015
Earlier work this paper cites.
LeCun, Y., Bengio, Y. and Hinton, G., 2015. Deep learning. Nature, 521(7553), pp.436-444
2015
Earlier work this paper cites.
Schmidhuber, Jürgen. ”Deep learning in neural networks: An overview.” Neural networks 61 (2015): 85-117
2015
Earlier work this paper cites.
Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M. and Tang, P.T.P., 2016. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836
Original
2016
Earlier work this paper cites.
Chen, J., Monga, R., Bengio, S. and Jozefowicz, R., 2016. Revisiting distributed synchronous SGD. arXiv preprint arXiv:1604.00981
Original
2016
Earlier work this paper cites.
He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778)
2016
Earlier work this paper cites.

Similar

Loshchilov, I. and Hutter, F., 2016. SGDR: stochastic gradient descent with restarts. arXiv preprint arXiv:1608.03983
Original
2016
Cited alongside, same era.
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y. and He, K., 2017. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint arXiv:1706.02677
Original
2017
Cited alongside, same era.
Cho, M., Finkler, U., Kumar, S., Kung, D., Saxena, V. and Sreedhar, D., 2017. PowerAI DDL. arXiv preprint arXiv:1708.02188
Original
2017
Cited alongside, same era.
You, Y., Zhang, Z., Hsieh, C.J. and Demmel, J., 2017. 100-epoch ImageNet Training with AlexNet in 24 Minutes. arXiv preprint arXiv:1709.05011
Original
2017
Cited alongside, same era.
Kurth, T., Zhang, J., Satish, N., Mitliagkas, I., Racah, E., Patwary, M.A., Malas, T., Sundaram, N., Bhimji, W., Smorkalov, M. and Deslippe, J., 2017. Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data. arXiv preprint arXiv:1708.05256
Original
2017
Cited alongside, same era.
Smith, L.N. and Topin, N., 2017. Exploring loss function topology with cyclical learning rates. arXiv preprint arXiv:1702.04283
Original
2017
Cited alongside, same era.
Sun, C., Shrivastava, A., Singh, S. and Gupta, A., 2017. Revisiting unreasonable effectiveness of data in deep learning era. arXiv preprint arXiv:1707.02968
Original
2017
Cited alongside, same era.
Zheng, S., Meng, Q., Wang, T., Chen, W., Yu, N., Ma, Z.M. and Liu, T.Y., 2017, July. Asynchronous Stochastic Gradient Descent with Delay Compensation. In International Conference on Machine Learning
2017
Cited alongside, same era.
Intel distribution of Caffe. https://github.com/intel/caffe, 2017
2017
Cited alongside, same era.
Intel Machine Learning Scaling Library for Linux OS. https://github.com/01org/MLSL, 2017
2017
Cited alongside, same era.

Then

Hu, J., Shen, L. and Sun, G., 2017. Squeeze-and-Excitation Networks. arXiv preprint arXiv:1709.01507
Original
2017
Closest in time.
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaev, O., Venkatesh, G. and Wu, H., 2017. Mixed Precision Training. arXiv preprint arXiv:1710.03740
Original
2017
Closest in time.
Silver, David, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert et al. ”Mastering the game of Go without human knowledge.” Nature 550, no. 7676 (2017): 354-359
2017
Closest in time.
Hoffer, E., Hubara, I. and Soudry, D., 2017. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741
Original
2017
Closest in time.
Smith, S.L., Kindermans, P.J. and Le, Q.V., 2017. Don’t Decay the Learning Rate, Increase the Batch Size. arXiv preprint arXiv:1711.00489
Original
2017
Closest in time.
Szegedy, C., Ioffe, S., Vanhoucke, V. and Alemi, A.A., 2017. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. In AAAI (pp. 4278-4284)
2017
Closest in time.
Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J.E. and Weinberger, K.Q., 2017. Snapshot ensembles: Train 1, get m for free. arXiv preprint arXiv:1704.00109
Original
2017
Closest in time.
Dinh, L., Pascanu, R., Bengio, S. and Bengio, Y., 2017. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933
Original
2017
Closest in time.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A. and Torralba, A., 2017. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence
2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…