2019

The OoO VLIW JIT Compiler for GPU Inference

Jain, Paras, Mo, Xiangxi, Jain, Ajay et al.

Understand

Current trends in Machine Learning~(ML) inference on hardware accelerated devices (e.g., GPUs, TPUs) point to alarmingly low utilization.

As ML inference is increasingly time-bounded by tight latency SLOs, increasing data parallelism is not an option.
The need for better efficiency motivates GPU multiplexing.
Furthermore, existing GPU programming abstractions force programmers to micro-manage GPU resources in an early-binding, context-free fashion.

Built on

The UNIX time-sharing system
Dennis M Ritchie and Ken Thompson. 1978 · 1929
Earlier work this paper cites.
Hitting the memory wall: implications of the obvious
Wm A Wulf and Sally A McKee. 1995 · 1995
Earlier work this paper cites.
Cellular Disco: Resource management using virtual clusters on shared-memory multiprocessors. In ACM SIGOPS Operating Systems Review , Vol. 33. ACM, 154–169
Kinshuk Govil, Dan Teodosiu, Yongqiang Huang, and Mendel Rosenblum. 1999 · 1999
Earlier work this paper cites.
MapReduce: simplified data processing on large clusters
Jeffrey Dean and Sanjay Ghemawat. 2008 · 2008
Earlier work this paper cites.
A view of the parallel computing landscape
Krste Asanovic, Rastislav Bodik, James Demmel, Tony Keaveny, Kurt Keutzer, John Kubiatowicz, Nelson Morgan, David Patterson, Koushik Sen, John Wawrzynek, et al · 2009
Earlier work this paper cites.
The Cost of Latency
James Hamilton. 2009 · 2009
Earlier work this paper cites.
Scaling the bandwidth wall: challenges in and avenues for CMP scaling. In ACM SIGARCH Computer Architecture News , Vol. 37. ACM, 371–382
Brian M Rogers, Anil Krishna, Gordon B Bell, Ken Vu, Xiaowei Jiang, and Yan Solihin. 2009 · 2009
Earlier work this paper cites.
Roofline: an insightful visual performance model for multicore architectures
Samuel Williams, Andrew Waterman, and David Patterson. 2009 · 2009
Earlier work this paper cites.
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center.. In NSDI , Vol. 11. 22–22
Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D Joseph, Randy H Katz, Scott Shenker, and Ion Stoica. 2011 · 2011
Earlier work this paper cites.
NVIDIA Kepler GK110 whitepaper
2012 · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems . 1097–1105
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012 · 2012
Earlier work this paper cites.
Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation . USENIX Association, 2–2
Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J Franklin, Scott Shenker, and Ion Stoica. 2012 · 2012
Earlier work this paper cites.
Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines
Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. 2013 · 2013
Earlier work this paper cites.

Similar

Very Deep Convolutional Networks for Large-Scale Image Recognition
Original
K. Simonyan and A. Zisserman. 2014 · 2014
Cited alongside, same era.
Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770–778
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 · 2016
Cited alongside, same era.
NVIDIA TensorRT
2017 · 2017
Cited alongside, same era.
The future is here: iPhone X
2018 · 2017
Cited alongside, same era.
Efficient Methods and Hardware for Deep Learning. In Proceedings of the Workshop on Trends in Machine-Learning (and Impact on Computer Architecture) (TiML ’17) . ACM, New York, NY, USA, Article 2
Bill Dally. 2017 · 2017
Cited alongside, same era.
Recent Advances in Artificial Intelligence via Machine Learning and the Implications for Computer System Design. In 2017 IEEE Hot Chips 29 Symposium
Jeff Dean. 2017 · 2017
Cited alongside, same era.
Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. 2017 · 2017
Cited alongside, same era.
In-datacenter performance analysis of a tensor processing unit. In Computer Architecture (ISCA), 2017 ACM/IEEE 44th Annual International Symposium on . IEEE, 1–12
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al · 2017
Cited alongside, same era.
AMD MxGPU: Hardware-based virtualization
2018 · 2018
Cited alongside, same era.
Edge TPU - Run Inference at the Edge | Edge TPU | Google Cloud
2018 · 2018
Cited alongside, same era.
Jetson AGX Xavier Developer Kit
2018 · 2018
Cited alongside, same era.
Kirin 980
2018 · 2018
Cited alongside, same era.

Then

InferLine: ML Inference Pipeline Composition Framework
Original
Daniel Crankshaw, Gur-Eyal Sela, Corey Zumar, Xiangxi Mo, Joseph E. Gonzalez, Ion Stoica, and Alexey Tumanov. 2018 · 2018
Later among the works it cites.
AWS Inferentia Machine Learning Processor
James Hamilton. 2018 · 2018
Later among the works it cites.
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective. In High Performance Computer Architecture (HPCA), 2018 IEEE International Symposium on . IEEE, 620–629
Kim Hazelwood, Sarah Bird, David Brooks, Soumith Chintala, Utku Diril, Dmytro Dzhulgakov, Mohamed Fawzy, Bill Jia, Yangqing Jia, Aditya Kalro, et al · 2018
Later among the works it cites.
Squeeze-and-Excitation Networks
Jie Hu, Li Shen, and Gang Sun. 2018 · 2018
Later among the works it cites.
Dynamic Space-Time Scheduling for GPU Inference
Original
Paras Jain, Xiangxi Mo, Ajay Jain, Harikaran Subbaraj, Rehan Sohail Durrani, Alexey Tumanov, Joseph Gonzalez, and Ion Stoica. 2018 · 2018
Later among the works it cites.
Amazon AWS ReInvent Keynote
Andy Jassy. 2018 · 2018
Later among the works it cites.
Tesla is building its own AI chips for self-driving cars
Greg Kumparak. 2018 · 2018
Later among the works it cites.
Accelerating Model Search with Model Batching
Deepak Narayanan, Keshav Santhanam, and Matei Zaharia. 2018 · 2018
Later among the works it cites.
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
Original
Jongsoo Park, Maxim Naumov, Protonu Basu, Summer Deng, Aravind Kalaiah, Daya Shanker Khudia, James Law, Parth Malani, Andrey Malevich, Nadathur Satish, Juan Pino, Martin Schatz, Alexander Sidorov, Viswanath Sivakumar, Andrew Tulloch, Xiaodong Wang, Yiming Wu, Hector Yuen, Utku Diril, Dmytro Dzhulgakov, Kim M. Hazelwood, Bill Jia, Yangqing Jia, Lin Qiao, Vijay Rao, Nadav Rotem, Sungjoo Yoo, and Mikhail Smelyanskiy. 2018 · 2018
Later among the works it cites.
Glow: Graph Lowering Compiler Techniques for Neural Networks
Original
Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Summer Deng, Roman Dzhabarov, James Hegeman, Roman Levenstein, Bert Maher, Nadathur Satish, Jakob Olesen, Jongsoo Park, Artem Rakhov, and Misha Smelyanskiy. 2018 · 2018
Later among the works it cites.
Introducing the Graphcore Rackscale IPU-POD
Ola Tørudbakken. 2018 · 2018
Later among the works it cites.
Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
Original
Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, William S. Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen. 2018 · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…