2020

Efficient Memory Management for Deep Neural Net Inference

Pisarchyk, Yury, Lee, Juhyun

Understand

While deep neural net inference was considered a task for servers only, latest advances in technology allow the task of inference to be moved to mobile and embedded devices, desired for various reasons ranging from latency to privacy.

These devices are not only limited by their compute power and battery, but also by their inferior physical memory and cache, and thus, an efficient memory manager becomes a crucial component for deep neural net inference at the edge.
We explore various strategies to smartly share memory buffers among intermediate tensors in deep neural nets.
Employing these can result in up to 11% smaller memory footprint than the state of the art.

Built on

Complete Register Allocation Problems
Sethi, R · 1975
Earlier work this paper cites.
Long short-term memory
Hochreiter, S. and Schmidhuber, J · 1997
Earlier work this paper cites.
Theano: A cpu and gpu math expression compiler
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y · 2010
Earlier work this paper cites.
Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z · 2015
Earlier work this paper cites.
Posenet: A convolutional network for real-time 6-dof camera relocalization
Kendall, A., Grimes, M., and Cipolla, R · 2015
Earlier work this paper cites.
Training deep nets with sublinear memory cost
Original
Chen, T., Xu, B., Zhang, C., and Guestrin, C · 2016
Earlier work this paper cites.

Similar

Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
Optimizing memory efficiency for deep convolutional neural networks on gpus
Li, C., Yang, Y., Feng, M., Chakradhar, S., and Zhou, H · 2016
Cited alongside, same era.
Rethinking the inception architecture for computer vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z · 2016
Cited alongside, same era.
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L · 2017
Cited alongside, same era.
Mobilenets: Efficient convolutional neural networks for mobile vision applications
Original
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H · 2017
Cited alongside, same era.

Then

Mobilenetv2: Inverted residuals and linear bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C · 2018
Later among the works it cites.
Profile-guided memory optimization for deep neural networks
Original
Sekiyama, T., Imamichi, T., Imai, H., and Raymond, R · 2018
Later among the works it cites.
Blazeface: Sub-millisecond neural face detection on mobile gpus
Original
Bazarevsky, V., Kartynnik, Y., Vakunov, A., Raveendran, K., and Grundmann, M · 2019
Later among the works it cites.
On-device neural net inference with mobile gpus
Lee, J., Chirkov, N., Ignasheva, E., Pisarchyk, Y., Shieh, M., Riccardi, F., Sarokin, R., Kulik, A., and Grundmann, M · 2019
Later among the works it cites.
Machine learning at facebook: Understanding inference at the edge
Original
Wu, C.-J., Brooks, D., Chen, K., Chen, D., Choudhury, S., Dukhan, M., Hazelwood, K., Isaac, E., Jia, Y., Jia, B., et al · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…