SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models

Understand

Large Language Models (LLMs) have become pivotal in advancing the field of artificial intelligence, yet their immense sizes pose significant challenges for both fine-tuning and deployment.

• Current post-training pruning methods, while reducing the sizes of LLMs, often fail to maintain their original performance.
• To address these challenges, this paper introduces SPP, a Sparsity-Preserved Parameter-efficient fine-tuning method.
• Different from existing post-training pruning approaches that struggle with performance retention, SPP proposes to employ lightweight learnable column and row matrices to optimize sparse LLM weights, keeping the structure and sparsity of pruned pre-trained models intact.