A Survey of Multi-Tenant Deep Learning Inference on GPU

Understand

Deep Learning (DL) models have achieved superior performance.

• Meanwhile, computing hardware like NVIDIA GPUs also demonstrated strong computing scaling trends with 2x throughput and memory bandwidth for each generation.
• With such strong computing scaling of GPUs, multi-tenant deep learning inference by co-locating multiple DL models onto the same GPU becomes widely deployed to improve resource utilization, enhance serving throughput, reduce energy cost, etc.
• However, achieving efficient multi-tenant DL inference is challenging which requires thorough full-stack system optimization.