2019

Benchmarking TPU, GPU, and CPU Platforms for Deep Learning

Wang, Yu Emma, Wei, Gu-Yeon, Brooks, David

Understand

Training deep learning models is compute-intensive and there is an industry-wide trend towards hardware specialization to improve performance.

To systematically benchmark deep learning platforms, we introduce ParaDnn, a parameterized benchmark suite for deep learning that generates end-to-end models for fully connected (FC), convolutional (CNN), and recurrent (RNN) neural networks.
Along with six real-world models, we benchmark Google's Cloud TPU v2/v3, NVIDIA's V100 GPU, and an Intel Skylake CPU platform.
We take a deep dive into TPU architecture, reveal its bottlenecks, and highlight valuable lessons learned for future specialized system design.

Built on

H.-T. Kung, “Why systolic architectures?” IEEE computer , vol. 15, no. 1, pp. 37–46, 1982
1982
Earlier work this paper cites.
D. Stroobandt, P. Verplaetse, and J. Van Campenhout, “Generating synthetic benchmark circuits for evaluating CAD tools,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , vol. 19, no. 9, pp. 1011–1022, 2000
2000
Earlier work this paper cites.
R. V. Hogg, J. McKean, and A. T. Craig, Introduction to mathematical statistics . Pearson Education, 2005
2005
Earlier work this paper cites.
J. L. Henning, “SPEC CPU2006 benchmark descriptions,” ACM SIGARCH Computer Architecture News , vol. 34, no. 4, pp. 1–17, 2006
2006
Earlier work this paper cites.
C. Bienia, S. Kumar, J. P. Singh, and K. Li, “The PARSEC benchmark suite: Characterization and architectural implications,” in Proceedings of the 17th international conference on Parallel architectures and compilation techniques . ACM, 2008, pp. 72–81
2008
Earlier work this paper cites.
A. Joshi, L. Eeckhout, and L. John, “The return of synthetic benchmarks,” in 2008 SPEC Benchmark Workshop , 2008, pp. 1–11
2008
Earlier work this paper cites.
S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer, S.-H. Lee, and K. Skadron, “Rodinia: A benchmark suite for heterogeneous computing,” in Workload Characterization, 2009. IISWC 2009. IEEE International Symposium on . Ieee, 2009, pp. 44–54
2009
Earlier work this paper cites.
S. Williams, A. Waterman, and D. Patterson, “Roofline: an insightful visual performance model for multicore architectures,” Communications of the ACM , vol. 52, no. 4, pp. 65–76, 2009
2009
Earlier work this paper cites.
K. Ganesan, J. Jo, W. L. Bircher, D. Kaseridis, Z. Yu, and L. K. John, “System-level max power (SYMPO)-a systematic approach for escalating system-level power consumption using synthetic benchmarks,” in Parallel Architectures and Compilation Techniques (PACT), 2010 19th International Conference on . IEEE, 2010, pp. 19–28
2010
Earlier work this paper cites.
Q. He, S. Zhou, B. Kobler, D. Duffy, and T. McGlynn, “Case study for running HPC applications in public clouds,” in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing . ACM, 2010, pp. 395–401
2010
Earlier work this paper cites.
J. Nickolls and W. J. Dally, “The GPU computing era,” IEEE Micro , vol. 30, no. 2, 2010
2010
Earlier work this paper cites.
J. L. Hennessy and D. A. Patterson, Computer architecture: a quantitative approach . Elsevier, 2011
2011
Earlier work this paper cites.
K. Kothari, “Comparison of several cloud computing providers,” Elixir Comp. Sci. & Engg , 2011
2011
Earlier work this paper cites.
T. Schaffter, D. Marbach, and D. Floreano, “GeneNetWeaver: in silico benchmark generation and performance profiling of network inference methods,” Bioinformatics , vol. 27, no. 16, pp. 2263–2270, 2011
2011
Earlier work this paper cites.
T. Chen, Y. Chen, M. Duranton, Q. Guo, A. Hashmi, M. Lipasti, A. Nere, S. Qiu, M. Sebag, and O. Temam, “BenchNN: On the broad potential application scope of hardware neural network accelerators,” in Workload Characterization (IISWC), 2012 IEEE International Symposium on . IEEE, 2012, pp. 36–45
2012
Earlier work this paper cites.
J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al. , “Large scale distributed deep networks,” in Advances in neural information processing systems , 2012, pp. 1223–1231
2012
Earlier work this paper cites.
D. Patterson, “For better or worse, benchmarks shape a field,” Communications of the ACM , vol. 55, 2012
2012
Earlier work this paper cites.
M. Turki, H. Mehrez, Z. Marrakchi, and M. Abid, “Towards synthetic benchmarks generator for CAD tool evaluation,” in Ph. D. Research in Microelectronics and Electronics (PRIME), 2012 8th Conference on . VDE, 2012, pp. 1–4
2012
Earlier work this paper cites.
K. Kim, C. Lee, J. H. Jung, and W. W. Ro, “Workload synthesis: Generating benchmark workloads from statistical execution profile,” in Workload Characterization (IISWC), 2014 IEEE International Symposium on . IEEE, 2014, pp. 120–129
2014
Earlier work this paper cites.

Similar

S. Thomas, C. Gohkale, E. Tanuwidjaja, T. Chong, D. Lau, S. Garcia, and M. B. Taylor, “CortexSuite: A synthetic brain benchmark suite.” in IISWC , 2014, pp. 76–79
2014
Cited alongside, same era.
S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding,” arXiv preprint arXiv:1510.00149 , 2015
Original
2015
Cited alongside, same era.
J. Hauswald, Y. Kang, M. A. Laurenzano, Q. Chen, C. Li, T. Mudge, R. G. Dreslinski, J. Mars, and L. Tang, “DjiNN and Tonic: DNN as a service and its implications for future warehouse scale computers,” in Computer Architecture (ISCA), 2015 ACM/IEEE 42nd Annual International Symposium on . IEEE, 2015, pp. 27–40
2015
Cited alongside, same era.
J. Hauswald, M. A. Laurenzano, Y. Zhang, C. Li, A. Rovinski, A. Khurana, R. G. Dreslinski, T. Mudge, V. Petrucci, L. Tang et al. , “Sirius: An open end-to-end voice and vision personal assistant and its implications for future warehouse scale computers,” in the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) , vol. 50, no. 4. ACM, 2015, pp. 223–238
2015
Cited alongside, same era.
M. Saleem, Q. Mehmood, and A.-C. N. Ngomo, “Feasible: A feature-based sparql benchmark generation framework,” in International Semantic Web Conference . Springer, 2015, pp. 52–69
2015
Cited alongside, same era.
R. Adolf, S. Rama, B. Reagen, G.-Y. Wei, and D. Brooks, “Fathom: Reference workloads for modern deep learning methods,” in Workload Characterization (IISWC), 2016 IEEE International Symposium on . IEEE, 2016, pp. 1–10
2016
Cited alongside, same era.
D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen et al. , “Deep speech 2: End-to-end speech recognition in english and mandarin,” in International Conference on Machine Learning , 2016, pp. 173–182
2016
Cited alongside, same era.
S. Bahrampour, N. Ramakrishnan, L. Schott, and M. Shah, “Comparative study of Caffe, Neon, Theano, and Torch for deep learning,” in ICLR , 2016
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , 2016, pp. 770–778
2016
Cited alongside, same era.
F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <0.5 mb model size,” arXiv preprint arXiv:1602.07360 , 2016
Original
2016
Cited alongside, same era.
S. Shi, Q. Wang, P. Xu, and X. Chu, “Benchmarking state-of-the-art deep learning software tools,” in Cloud Computing and Big Data (CCBD), 2016 7th International Conference on . IEEE, 2016, pp. 99–104
2016
Cited alongside, same era.
Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al. , “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144 , 2016
Original
2016
Cited alongside, same era.

Then

Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” arXiv preprint arXiv:1712.01887 , 2017
Original
2017
Later among the works it cites.
B. Research, “Deepbench https://github.com/baidu-research/DeepBench ,” 2017
2017
Later among the works it cites.
D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton et al. , “Mastering the game of Go without human knowledge,” Nature , vol. 550, no. 7676, p. 354, 2017
2017
Later among the works it cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , 2017, pp. 5998–6008
2017
Later among the works it cites.
“TensorFlow: Using JIT compilation https://www.tensorflow.org/xla/jit ,” 2018
2018
Later among the works it cites.
“ https://cloud.google.com/tpu/docs/system-architecture ,” Google Cloud Documentation , 2018
2018
Later among the works it cites.
R. Banner, I. Hubara, E. Hoffer, and D. Soudry, “Scalable methods for 8-bit training of neural networks,” arXiv preprint arXiv:1805.11046 , 2018
Original
2018
Later among the works it cites.
L. Case, “Volta Tensor Core GPU achieves new AI performance milestones,” Nvidia Developer Blog , 2018
2018
Later among the works it cites.
T. Chen, T. Moreau, Z. Jiang, H. Shen, E. Yan, L. Wang, Y. Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy, “TVM: End-to-end optimization stack for deep learning,” arXiv preprint arXiv:1802.04799 , 2018
Original
2018
Later among the works it cites.
K. Hazelwood, S. Bird, D. Brooks, S. Chintala, U. Diril, D. Dzhulgakov, M. Fawzy, B. Jia, Y. Jia, A. Kalro, J. Law, K. Lee, J. Lu, P. Noordhuis, M. Smelyanskiy, L. Xiong, and X. Wang, “Applied machine learning at Facebook: A datacenter infrastructure perspective,” in High Performance Computer Architecture (HPCA), 2018 IEEE International Symposium on . IEEE, 2018, pp. 620–629
2018
Later among the works it cites.
Z. Jia, M. Zaharia, and A. Aiken, “Beyond data and model parallelism for deep neural networks,” arXiv preprint arXiv:1807.05358 , 2018
Original
2018
Later among the works it cites.
——, “MLPerf: SPEC for ML,” https://rise.cs.berkeley.edu/blog/mlperf-spec-for-ml/ , 2018
2018
Later among the works it cites.
T. repository for TPU models, “ https://github.com/tensorflow/tpu ,” Github , 2018
2018
Later among the works it cites.
N. Rotem, J. Fix, S. Abdulrasool, S. Deng, R. Dzhabarov, J. Hegeman, R. Levenstein, B. Maher, S. Nadathur, J. Olesen et al. , “Glow: Graph lowering compiler techniques for neural networks,” arXiv preprint arXiv:1805.00907 , 2018
Original
2018
Later among the works it cites.
C. J. Shallue, J. Lee, J. Antognini, J. Sohl-Dickstein, R. Frostig, and G. E. Dahl, “Measuring the effects of data parallelism on neural network training,” arXiv preprint arXiv:1811.03600 , 2018
Original
2018
Later among the works it cites.
J.-H. Tao, Z.-D. Du, Q. Guo, H.-Y. Lan, L. Zhang, S.-Y. Zhou, L.-J. Xu, C. Liu, H.-F. Liu, S. Tang, W. Chen, S.-L. Liu, and Y.-J. Chen, “BenchIP: Benchmarking intelligence processors,” Journal of Computer Science and Technology , vol. 33, no. 1, pp. 1–23, 2018
2018
Later among the works it cites.
W. Wei, L. Xu, L. Jin, W. Zhang, and T. Zhang, “AI matrix-synthetic benchmarks for DNN,” arXiv preprint arXiv:1812.00886 , 2018
Original
2018
Later among the works it cites.
G. A. Blog, “Introducing GPipe, an open source library for efficiently training large-scale neural network models,” https://ai.googleblog.com/2019/03/introducing-gpipe-open-source-library.html , 2019
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…