Inference-Time Hyper-Scaling with KV Cache Compression

Understand

Inference-time scaling trades efficiency for increased reasoning accuracy by generating longer or more parallel sequences.

• However, in Transformer LLMs, generation cost is bottlenecked by the size of the key-value (KV) cache, rather than the number of generated tokens.
• Hence, we explore inference-time hyper-scaling: by compressing the KV cache, we can generate more tokens within the same compute budget and further improve the accuracy of scaled inference.
• The success of this approach, however, hinges on the ability of compression methods to preserve accuracy even at high compression ratios.