CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering

Understand

Large language models (LLMs) with extended context windows have become increasingly prevalent for tackling complex tasks.

• However, the substantial Key-Value (KV) cache required for long-context LLMs poses significant deployment challenges.
• Existing approaches either discard potentially critical information needed for future generations or offer limited efficiency gains due to high computational overhead.
• In this paper, we introduce CentroidKV, a simple yet effective framework for online KV cache clustering.