INFaaS: A Model-less and Managed Inference Serving System

Understand

Despite existing work in machine learning inference serving, ease-of-use and cost efficiency remain challenges at large scales.

• Developers must manually search through thousands of model-variants -- versions of already-trained models that differ in hardware, resource footprints, latencies, costs, and accuracies -- to meet the diverse application requirements.
• Since requirements, query load, and applications themselves evolve over time, these decisions need to be made dynamically for each inference query to avoid excessive costs through naive autoscaling.
• To avoid navigating through the large and complex trade-off space of model-variants, developers often fix a variant across queries, and replicate it when load increases.