Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training

Understand

Small language models (SLMs) have attracted considerable attention from both academia and industry due to their broad range of applications in edge devices.

• To obtain SLMs with strong performance, conventional approaches either pre-train the models from scratch, which incurs substantial computational costs, or compress/prune existing large language models (LLMs), which results in performance drops and falls short in comparison to pre-training.
• In this paper, we investigate the family of acceleration methods that involve both structured pruning and model training.
• We found 1) layer-wise adaptive pruning (Adapt-Pruner) is extremely effective in LLMs and yields significant improvements over existing pruning techniques, 2) adaptive pruning equipped with further training leads to models comparable to those pre-training from scratch, 3) incremental pruning brings non-trivial performance gain by interleaving pruning with training and only removing a small portion of neurons ($\sim$5%) at a time.