CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective

Understand

Large language models have revolutionized natural language processing but face significant challenges of high storage and runtime costs, due to the transformer architecture's reliance on self-attention, particularly the large KV cache for long-sequence inference.

• Recent efforts to reduce KV cache size by pruning less critical entries based on attention weights remain empirical and lack formal grounding.
• This paper presents a formal study on identifying critical KV cache entries by analyzing attention output perturbation.
• Our analysis reveals that, beyond attention weights, the value states within KV entries and pretrained parameter matrices are also crucial.