BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching

Understand

Large language models (LLMs) increasingly play an important role in a wide range of information processing and management tasks in industry.

• Many of these tasks are performed in large batches or even offline, and the performance indicator for which is throughput.
• These tasks usually show the characteristic of prefix sharing, where different prompt input can partially show the common prefix.
• However, the existing LLM inference engines tend to optimize the streaming requests and show limitations of supporting the large batched tasks with the prefix sharing characteristic.