ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production

Understand

With the widespread adoption of Large Language Models (LLMs), serving LLM inference requests has become an increasingly important task, attracting active research advancements.

• Practical workloads play an essential role in this process: they are critical for motivating and benchmarking serving techniques and systems.
• However, the existing understanding of real-world LLM serving workloads is limited due to the lack of a comprehensive workload characterization.
• Prior analyses remain insufficient in scale and scope, thus failing to fully capture intricate workload characteristics.