2019

The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study

Tang, Zhenheng, Wang, Yuxin, Wang, Qiang et al.

Understand

Over the past years, great progress has been made in improving the computing power of general-purpose graphics processing units (GPGPUs), which facilitates the prosperity of deep neural networks (DNNs) in multiple fields like computer vision and natural language processing.

A typical DNN training process repeatedly updates tens of millions of parameters, which not only requires huge computing resources but also consumes significant energy.
In order to train DNNs in a more energy-efficient way, we empirically investigate the impact of GPU Dynamic Voltage and Frequency Scaling (DVFS) on the energy consumption and performance of deep learning.
Our experiments cover a wide range of GPU architectures, DVFS settings, and DNN configurations.

Built on

K. Chellapilla, S. Puri, and P. Simard, “High Performance Convolutional Neural Networks for Document Processing,” in Tenth International Workshop on Frontiers in Handwriting Recognition, La Baule, France, October, 2006 , 2006
2006
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, Florida, USA, June 2009, , June 2009
2009
Earlier work this paper cites.
Y. Jiao, H. Lin, P. Balaji, and W. Feng, “Power and performance characterization of computational kernels on the gpu,” in In International Conference on Green Computing and Communications, Hangzhou, China, December, 2010 , 2010
2010
Earlier work this paper cites.
J. Lee, V. Sathisha, M. J. Schulte, K. Compton, and N. Kim, “Improving Throughput of Power-Constrained GPUs Using Dynamic Voltage/Frequency and Core Scaling,” in 2011 International Conference on PACT, Galveston, TX, USA, October, 2011 , 2011
2011
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Neural Information Processing Systems (NeurIPS), Miami, Florida, USA, June, 2009 , 2012
2012
Earlier work this paper cites.
J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, and Q. V. Le, “Large scale distributed deep networks,” in Advances in Neural Information Processing Systems (NeurIPS), Lake Tahoe, Nevada, United States, December, 2012 , 2012
2012
Earlier work this paper cites.
R. Ge, R. Vogt, J. Majumder, A. Alam, M. Burtscher, and Z. Zong, “Effects of dynamic voltage and frequency scaling on a k20 gpu,” in 2013 42nd International Conference on Parallel Processing, Lyon, France, October, 2013 , 2013
2013
Earlier work this paper cites.
M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B. Su, “Scaling distributed machine learning with the parameter server,” in 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI ’14, Broomfield, CO, USA, October, 2014. , 2014
2014
Earlier work this paper cites.
S. Lee, J. K. Kim, X. Zheng, Q. Ho, G. A. Gibson, and E. P. Xing, “On model parallelization and scheduling strategies for distributed machine learning,” in Advances in Neural Information Processing Systems (NeurIPS), Montreal, Quebec, Canada, December, 2014 , 2014
2014
Earlier work this paper cites.
X. Mei, K. Zhao, C. Liu, and X. Chu, “Benchmarking the memory hierarchy of modern GPUs,” in Network and Parallel Computing - 11th IFIP, Ilan, Taiwan, September, 2014 , 2014
2014
Earlier work this paper cites.
M. Michaël, M. Henaff, and Y. LeCun, “Fast training of convolutional networks through ffts,” in 2nd International Conference on Learning Representations (ICLR), Banff, AB, Canada, April, 2014 , 2014
2014
Earlier work this paper cites.
S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cudnn: Efficient primitives for deep learning,” arXiv preprint arXiv:1410.0759 , 2014
Original
2014
Earlier work this paper cites.
Y. Abe, H. Sasaki, S. Kato, K. Inoue, M. Edahiro, and M. Peres, “Power and performance characterization and modeling of gpu-accelerated systems,” in IEEE 28th International Parallel and Distributed Processing Symposium (IPDPS), Phoenix, AZ, USA, May, 2014 , 2014
2014
Earlier work this paper cites.
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA, November, 2014 , 2014
2014
Earlier work this paper cites.
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature , vol. 521, no. 7553, pp. 436–444, 2015
2015
Earlier work this paper cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, June, 2015 , June 2015
2015
Earlier work this paper cites.

Similar

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA, May, 2015, , 2015
2015
Cited alongside, same era.
R. Girshick, “Fast r-cnn,” in 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, December, 2015 , December 2015
2015
Cited alongside, same era.
I. Paul, W. Huang, M. Arora, and S. Yalamanchili, “Harmonia: Balancing compute and memory power in high-performance gpus,” in 2015 ACM/IEEE 42nd Annual International Symposium on Computer Architecture (ISCA), Portland, OR, USA, June, 2015 , 2015
2015
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June, 2016 , June 2016
2016
Cited alongside, same era.
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in 14th European conference on computer vision (ECCV), Amsterdam, The Netherlands, October, 2016 , 2016
2016
Cited alongside, same era.
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June, 2016 , June 2016
2016
Cited alongside, same era.
D. Das, S. Avancha, D. Mudigere, K. Vaidynathan, S. Sridharan, D. Kalamkar, B. Kaul, and P. Dubey, “Distributed deep learning using synchronous stochastic gradient descent,” arXiv preprint arXiv:1602.06709 , 2016
Original
2016
Cited alongside, same era.
R. A. Bridges, N. Imam, and T. M. Mintz, “Understanding GPU Power: A Survey of Profiling, Modeling, and Simulation Methods,” ACM CSUR , vol. 49, no. 3, p. 41, 2016
2016
Cited alongside, same era.
A. Lavin and S. Gray, “Fast algorithms for convolutional neural networks,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June, 2016 , June 2016
2016
Cited alongside, same era.
X. Li, G. Zhang, H. H. Huang, Z. Wang, and W. Zheng, “Performance analysis of gpu-based convolutional neural networks,” in 2016 45th International Conference on Parallel Processing (ICPP), Philadelphia, PA, USA, August, 2016 , 2016
2016
Cited alongside, same era.
S. Shi, Q. Wang, P. Xu, and X. Chu, “Benchmarking state-of-the-art deep learning software tools,” in 2016 7th International Conference on Cloud Computing and Big Data (CCBD), Macau, China, November, 2016 , Nov 2016
2016
Cited alongside, same era.
D. Li, X. Chen, M. Becchi, and Z. Zong, “Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus,” in 2016 IEEE International Conferences on Sustainable Computing and Communications (SustainCom), Atlanta, GA, USA, October, 2016 , 2016
2016
Cited alongside, same era.

Then

X. Mei and X. Chu, “Dissecting GPU Memory Hierarchy Through Microbenchmarking,” IEEE TPDS , vol. 28, no. 1, pp. 72–86, Jan 2017
2017
Later among the works it cites.
V. Sze, Y. Chen, T. Yang, and J. S. Emer, “Efficient processing of deep neural networks: A tutorial and survey,” Proceedings of the IEEE , vol. 105, no. 12, pp. 2295–2329, Dec 2017
2017
Later among the works it cites.
B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” in 5th International Conference on Learning Representations (ICLR), Toulon, France, April , 2017 , 2017
2017
Later among the works it cites.
Q. Wang and X. Chu, “GPGPU Power Estimation with Core and Memory Frequency Scaling,” SIGMETRICS Perform. Eval. Rev. , vol. 45, no. 2, pp. 73–78, Oct. 2017
2017
Later among the works it cites.
X. Mei, X. Chu, H. Liu, Y. Leung, and Z. Li, “Energy efficient real-time task scheduling on cpu-gpu hybrid clusters,” in 2017 IEEE Conference on Computer Communications (INFOCOM), Atlanta, GA, USA, May, 2017 , 2017
2017
Later among the works it cites.
V. Chau, X. Chu, H. Liu, and Y. Leung, “Energy efficient job scheduling with dvfs for cpu-gpu heterogeneous systems,” in Proceedings of the Eighth International Conference on Future Energy Systems (e-Energy ’17), Shatin, Hong Kong, China, May, 2017 , 2017
2017
Later among the works it cites.
S. Shams, R. Platania, K. Lee, and S. Park, “Evaluation of deep learning frameworks over different hpc architectures,” in 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), Atlanta, GA, USA, June, 2017 , June 2017
2017
Later among the works it cites.
E. Cai, D. Juan, D. Stamoulis, and D. Marculescu, “Neuralpower: Predict and deploy energy-efficient convolutional neural networks,” in Proceedings of The 9th Asian Conference on Machine Learning (ACML), Seoul, Korea, November, 2017 , 2017
2017
Later among the works it cites.
X. Jia, S. Song, W. He, Y. Wang, H. Rong, F. Zhou, L. Xie, Z. Guo, Y. Yang, and L. Yu, “Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes,” in NeurIPS Workshop on Systems for ML and Open Source Software, Montréal, Canada, December, 2018 , 2018
2018
Later among the works it cites.
Y. You, Z. Zhang, C. Hsieh, J. Demmel, and K. Keutzer, “Imagenet training in minutes,” in Proceedings of the 47th International Conference on Parallel Processing (ICPP), Eugene, OR, USA, August, 2018 , 2018
2018
Later among the works it cites.
Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” in 6th International Conference on Learning Representations (ICLR), BC, Canada, April 30 - May 3, 2018 , 2018
2018
Later among the works it cites.
C. Chen, J. Choi, D. Brand, A. Agrawal, W. Zhang, and K. Gopalakrishnan, “Adacomp: Adaptive residual gradient compression for data-parallel distributed training,” in Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), New Orleans, Louisiana, USA, February, 2018 , 2018
2018
Later among the works it cites.
Q. Wang and X. Chu, “Gpgpu performance estimation with core and memory frequency scaling,” in 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS), Singapore, December, 2018 , 2018
2018
Later among the works it cites.
S. Shi, Q. Wang, and X. Chu, “Performance modeling and evaluation of distributed deep learning frameworks on gpus,” in 4th Intl Conf on Big Data Intelligence and Computing(DataCom), Athens, Greece, August , 2018 , Aug 2018
2018
Later among the works it cites.
NVIDIA, “NVIDIA System Management Interface (nvidia-smi),” [Online] https://developer.nvidia.com/nvidia-system-management-interface, 2018
2018
Later among the works it cites.
NVIDIA, “NVIDIA Management Library ,” [Online] https://developer.nvidia.com/nvidia-management-library-nvml, 2018
2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…