2018

SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks

Wang, Linnan, Ye, Jinmian, Zhao, Yiyang et al.

Understand

Going deeper and wider in neural architectures improves the accuracy, while the limited GPU DRAM places an undesired restriction on the network design domain.

Deep Learning (DL) practitioners either need change to less desired network architectures, or nontrivially dissect a network across multiGPUs.
These distract DL practitioners from concentrating on their original machine learning tasks.
We present SuperNeurons: a dynamic GPU memory scheduling runtime to enable the network training far beyond the GPU DRAM capacity.

Built on

Second order derivatives for network pruning: Optimal brain surgeon
Hassibi, B., and Stork, D. G · 1993
Earlier work this paper cites.
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., and Frasconi, P · 1994
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P · 1998
Earlier work this paper cites.
Torch: a modular machine learning software library
Collobert, R., Bengio, S., and Mariéthoz, J · 2002
Earlier work this paper cites.
Spark: Cluster computing with working sets
Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., and Stoica, I · 2010
Earlier work this paper cites.
Improving the speed of neural networks on cpus
Vanhoucke, V., Senior, A., and Mao, M. Z · 2011
Earlier work this paper cites.
Large scale distributed deep networks
Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., et al · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Deep learning with cots hpc systems
Coates, A., Huval, B., Wang, T., Wu, D., Catanzaro, B., and Andrew, N · 2013
Earlier work this paper cites.

Similar

cudnn: Efficient primitives for deep learning
Original
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., and Shelhamer, E · 2014
Cited alongside, same era.
Caffe: Convolutional architecture for fast feature embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T · 2014
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K., and Zisserman, A · 2014
Cited alongside, same era.
Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems
Original
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z · 2015
Cited alongside, same era.
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A · 2015
Cited alongside, same era.
Hierarchical dag scheduling for hybrid distributed systems
Wu, W., Bouteiller, A., Bosilca, G., Faverge, M., and Dongarra, J · 2015
Cited alongside, same era.
Tensorflow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al · 2016
Cited alongside, same era.
Training deep nets with sublinear memory cost
Original
Chen, T., Xu, B., Zhang, C., and Guestrin, C · 2016
Cited alongside, same era.
Eie: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J · 2016
Cited alongside, same era.
http://mxnet.io/architecture/note_memory.html
Mxnet’s graph representation of neural networks
Cited in the paper.
Comparative study of caffe, neon, theano, and torch for deep learning
Bahrampour, S., Ramakrishnan, N., Schott, L., and Shah, M
Cited in the paper.

Then

Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Later among the works it cites.
Densely connected convolutional networks
Original
Huang, G., Liu, Z., Weinberger, K. Q., and van der Maaten, L · 2016
Later among the works it cites.
Proteus: Exploiting numerical precision variability in deep neural networks
Judd, P., Albericio, J., Hetherington, T., Aamodt, T. M., Jerger, N. E., and Moshovos, A · 2016
Later among the works it cites.
vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design
Rhu, M., Gimelshein, N., Clemons, J., Zulfiqar, A., and Keckler, S. W · 2016
Later among the works it cites.
Efficient communications in training large scale neural networks
Original
Wang, L., Wu, W., Bosilca, G., Vuduc, R., and Xu, Z · 2016
Later among the works it cites.
Blasx: A high performance level-3 blas library for heterogeneous multi-gpu computing
Wang, L., Wu, W., Xu, Z., Xiao, J., and Yang, Y · 2016
Later among the works it cites.
Memory-efficient implementation of densenets
Original
Pleiss, G., Chen, D., Huang, G., Li, T., van der Maaten, L., and Weinberger, K. Q · 2017
Later among the works it cites.
Inception-v4, inception-resnet and the impact of residual connections on learning
Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. A · 2017
Later among the works it cites.
Accelerating deep neural network training with inconsistent stochastic gradient descent
Wang, L., Yang, Y., Min, R., and Chakradhar, S · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…