2019

A Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learning

Ben-Nun, Tal, Besta, Maciej, Huber, Simon et al.

Understand

We introduce Deep500: the first customizable benchmarking infrastructure that enables fair comparison of the plethora of deep learning frameworks, algorithms, libraries, and techniques.

The key idea behind Deep500 is its modular design, where deep learning is factorized into four distinct levels: operators, network processing, training, and distributed training.
Our evaluation illustrates that Deep500 is customizable (enables combining and benchmarking different deep learning codes) and fair (uses carefully selected metrics).
Moreover, Deep500 is fast (incurs negligible overheads), verifiable (offers infrastructure to analyze correctness), and reproducible.

Built on

H. Robbins and S. Monro, “A stochastic approximation method,” The Annals of Mathematical Statistics , 1951
1951
Earlier work this paper cites.
J. J. Dongarra et al. , “Top500 supercomputer sites,” 1994
1994
Earlier work this paper cites.
Y. LeCun and C. Cortes, “The MNIST database of handwritten digits,” 1998
1998
Earlier work this paper cites.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE , 1998
1998
Earlier work this paper cites.
J. S. Vetter and M. O. McCracken, “Statistical scalability analysis of communication operations in distributed applications,” in Proceedings of the Eighth ACM SIGPLAN Symposium on Principles and Practices of Parallel Programming , ser. PPoPP ’01. New York, NY, USA: ACM, 2001, pp. 123–132
2001
Earlier work this paper cites.
R. Collobert, S. Bengio, and J. Mariéthoz, “Torch: a modular machine learning software library,” Idiap, Tech. Rep., 2002
2002
Earlier work this paper cites.
K. Chellapilla, S. Puri, and P. Simard, “High Performance Convolutional Neural Networks for Document Processing,” in ICFHR , 2006
2006
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR , 2009
2009
Earlier work this paper cites.
A. Krizhevsky, “Learning multiple layers of features from tiny images,” Master’s thesis, 2009
2009
Earlier work this paper cites.
J. Bergstra, F. Bastien, O. Breuleux, P. Lamblin, R. Pascanu, O. Delalleau, G. Desjardins, D. Warde-Farley, I. Goodfellow, A. Bergeron et al. , “Theano: Deep learning on GPUs with Python,” in NIPS, BigLearning Workshop, Granada, Spain , 2011
2011
Earlier work this paper cites.
B. Recht, C. Re, S. Wright, and F. Niu, “Hogwild: A lock-free approach to parallelizing stochastic gradient descent,” in NIPS , 2011
2011
Earlier work this paper cites.
J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al. , “Large scale distributed deep networks,” in Advances in neural information processing systems , 2012
2012
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25 , 2012, pp. 1097–1105
2012
Earlier work this paper cites.
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in ACMMM , 2014
2014
Earlier work this paper cites.
D. Yu, A. Eversole, M. Seltzer, K. Yao, Z. Huang, B. Guenter, O. Kuchaiev, Y. Zhang, F. Seide, H. Wang et al. , “An introduction to computational networks and the computational network toolkit,” Microsoft Technical Report MSR-TR-2014–112 , 2014
2014
Earlier work this paper cites.
T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang, “Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems,” arXiv:1512.01274 , 2015
Original
2015
Earlier work this paper cites.
S. Tokui, K. Oono, S. Hido, and J. Clayton, “Chainer: a next-generation open source framework for deep learning,” in LearningSys @NIPS , 2015
2015
Earlier work this paper cites.

Similar

F. Chollet, “Keras,” 2015
2015
Cited alongside, same era.
T. Hoefler and R. Belli, “Scientific benchmarking of parallel computing systems,” in SC , 2015
2015
Cited alongside, same era.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR , 2015
2015
Cited alongside, same era.
A. Lavin and S. Gray, “Fast algorithms for convolutional neural networks,” in CVPR , 2016
2016
Cited alongside, same era.
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al. , “Tensorflow: a system for large-scale machine learning.” in OSDI , 2016
2016
Cited alongside, same era.
D. Team et al. , “Deeplearning4j: Open-source distributed deep learning for the JVM,” Apache Software Foundation License , vol. 2, 2016
2016
Cited alongside, same era.
A. Damien and Others, “Tflearn,” 2016
2016
Cited alongside, same era.
R. Adolf, S. Rama, B. Reagen, G. Wei, and D. M. Brooks, “Fathom: Reference workloads for modern deep learning methods,” arXiv:1608.06581 , 2016
Original
2016
Cited alongside, same era.
P. Moritz, R. Nishihara, and M. I. Jordan, “A linearly-convergent stochastic l-bfgs algorithm,” in ICAIS , 2016
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR , 2016
2016
Cited alongside, same era.
S. Zagoruyko and N. Komodakis, “Wide residual networks,” arXiv:1605.07146 , 2016
Original
2016
Cited alongside, same era.
H. Dong, A. Supratak, L. Mai, F. Liu, A. Oehmichen, S. Yu, and Y. Guo, “Tensorlayer: a versatile library for efficient deep learning development,” in ACMMM , 2017
2017
Cited alongside, same era.

Then

J. Redmon, “Darknet: Open source neural networks in c,” 2013–2018
2018
Later among the works it cites.
T. Chen, T. Moreau, Z. Jiang, H. Shen, E. Q. Yan, L. Wang, Y. Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy, “TVM: end-to-end optimization stack for deep learning,” arXiv:1802.04799 , 2018
Original
2018
Later among the works it cites.
A. Sergeev and M. Del Balso, “Horovod: fast and easy distributed deep learning in tensorflow,” arXiv:1802.05799 , 2018
Original
2018
Later among the works it cites.
MLPerf, “A broad ML benchmark suite,” 2018. [Online]. Available: https://www.mlperf.org
2018
Later among the works it cites.
H. Zhu, M. Akrout, B. Zheng, A. Pelegris, A. Phanishayee, B. Schroeder, and G. Pekhimenko, “Tbd: Benchmarking and analyzing deep neural network training.” IISWC, 2018
2018
Later among the works it cites.
HPE, “Deep learning benchmarking suite,” 2018. [Online]. Available: https://developer.hpe.com/platform/hpe-deep-learning-cookbook/home
2018
Later among the works it cites.
Kaggle, “Data science competition repository,” 2018. [Online]. Available: https://www.kaggle.com
2018
Later among the works it cites.
B. Research, “Deepbench: Benchmarking deep learning operations on different hardware,” 2018. [Online]. Available: https://github.com/baidu-research/DeepBench
2018
Later among the works it cites.
X. Jia, S. Song, W. He, Y. Wang, H. Rong, F. Zhou, L. Xie, Z. Guo, Y. Yang, L. Yu, T. Chen, G. Hu, S. Shi, and X. Chu, “Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes,” 2018
2018
Later among the works it cites.
GitHub, “Deep learning repositories on github,” 2018. [Online]. Available: https://github.com/topics/deep-learning
2018
Later among the works it cites.
Open Neural Network Exchange, “Onnx github repository,” 2018. [Online]. Available: https://www.github.com/onnx
2018
Later among the works it cites.
Khronos Group, “Neural Network Exchange Format,” 2018
2018
Later among the works it cites.
K. Y. Levy, A. Yurtsever, and V. Cevher, “Online adaptive methods, universality and acceleration,” arXiv:1809.02864 , 2018
Original
2018
Later among the works it cites.
E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized Evolution for Image Classifier Architecture Search,” 2018
2018
Later among the works it cites.
Y. Oyama, T. Ben-Nun, T. Hoefler, and S. Matsuoka, “Accelerating deep learning frameworks with micro-batches,” in IEEE International Conference on Cluster Computing (CLUSTER) , 2018
2018
Later among the works it cites.
C. Renggli, D. Alistarh, and T. Hoefler, “Sparcml: High-performance sparse communication for machine learning,” arXiv:1802.08021 , 2018
Original
2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…