MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees

Understand

Open-weight large language model (LLM) zoos provide access to numerous high-quality models, but selecting the appropriate model for specific tasks remains challenging and requires technical expertise.

• Most users simply want factually correct, safe, and satisfying responses without concerning themselves with model technicalities, while inference service providers prioritize minimizing operating costs.
• These competing interests are typically mediated through service level agreements (SLAs) that guarantee minimum service quality.
• We introduce MESS+, a stochastic optimization algorithm for cost-optimal LLM request routing while providing rigorous SLA compliance guarantees.