SliceGPT: Compress Large Language Models by Deleting Rows and Columns

Understand

Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources.

• Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc.
• Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware.
• In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network.