CORM: Cache Optimization with Recent Message for Large Language Model Inference

Understand

Large Language Models (LLMs), despite their remarkable performance across a wide range of tasks, necessitate substantial GPU memory and consume significant computational resources.

• Beyond the memory taken up by model weights, the memory used by the KV cache rises linearly with sequence length, becoming a primary bottleneck for inference.
• In this paper, we introduce an innovative method for optimizing the KV cache, which considerably minimizes its memory footprint.
• Upon thorough investigation, we discover that in most Transformer models, (i) there is a striking similarity between adjacent tokens' query vectors, and (ii) the attention calculation of the current query can rely exclusively on the attention information of a small fraction of preceding queries.