Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfiguration

Understand

Continuous learning (CL) has emerged as one of the most popular deep learning paradigms deployed in modern cloud GPUs.

• Specifically, CL has the capability to continuously update the model parameters (through model retraining) and use the updated model (if available) to serve overtime arriving inference requests.
• It is generally beneficial to co-locate the retraining and inference together to enable timely model updates and avoid model transfer overheads.
• This brings the need for GPU sharing among retraining and inferences.