On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference

Understand

Despite the recent success associated with Large Language Models (LLMs), they are notably cost-prohibitive to deploy in resource-constrained environments due to their excessive memory and computational demands.

• In addition to model parameters, the key-value cache is also stored in GPU memory, growing linearly with batch size and sequence length.
• As a remedy, recent works have proposed various eviction policies for maintaining the overhead of key-value cache under a given budget.
• This paper embarks on the efficacy of existing eviction policies in terms of importance score calculation and eviction scope construction.