ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models

Understand

The impressive performance of Large Language Models (LLMs) across various natural language processing tasks comes at the cost of vast computational resources and storage requirements.

• One-shot pruning techniques offer a way to alleviate these burdens by removing redundant weights without the need for retraining.
• Yet, the massive scale of LLMs often forces current pruning approaches to rely on heuristics instead of optimization-based techniques, potentially resulting in suboptimal compression.
• In this paper, we introduce ALPS, an optimization-based framework that tackles the pruning problem using the operator splitting technique and a preconditioned conjugate gradient-based post-processing step.