Using Python for Model Inference in Deep Learning

Understand

Python has become the de-facto language for training deep neural networks, coupling a large suite of scientific computing libraries with efficient libraries for tensor computation such as PyTorch or TensorFlow.

• However, when models are used for inference they are typically extracted from Python as TensorFlow graphs or TorchScript programs in order to meet performance and packaging constraints.
• The extraction process can be time consuming, impeding fast prototyping.
• We show how it is possible to meet these performance and packaging constraints while performing inference in Python.