KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models

Understand

The bottleneck associated with the key-value(KV) cache presents a significant challenge during the inference processes of large language models.

• While depth pruning accelerates inference, it requires extensive recovery training, which can take up to two weeks.
• On the other hand, width pruning retains much of the performance but offers slight speed gains.
• To tackle these challenges, we propose KVPruner to improve model efficiency while maintaining performance.