PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs

Understand

Recently, significant progress has been made in developing reasoning-capable Large Language Models (LLMs) through long Chain-of-Thought (CoT) techniques.

• However, this long-CoT reasoning process imposes substantial memory overhead due to the large Key-Value (KV) Cache memory overhead.
• Post-training KV Cache quantization has emerged as a promising compression technique and has been extensively studied in short-context scenarios.
• However, directly applying existing methods to long-CoT LLMs causes significant performance degradation due to the following two reasons: (1) Large cumulative error: Existing methods fail to adequately leverage available memory, and they directly quantize the KV Cache during each decoding step, leading to large cumulative quantization error.