AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates

Understand

Structured weight pruning is a representative model compression technique of DNNs to reduce the storage and computation requirements and accelerate inference.

• An automatic hyperparameter determination process is necessary due to the large number of flexible hyperparameters.
• This work proposes AutoCompress, an automatic structured pruning framework with the following key performance improvements: (i) effectively incorporate the combination of structured pruning schemes in the automatic process; (ii) adopt the state-of-art ADMM-based structured weight pruning as the core algorithm, and propose an innovative additional purification step for further weight reduction without accuracy loss; and (iii) develop effective heuristic search method enhanced by experience-based guided search, replacing the prior deep reinforcement learning technique which has underlying incompatibility with the target pruning problem.
• Extensive experiments on CIFAR-10 and ImageNet datasets demonstrate that AutoCompress is the key to achieve ultra-high pruning rates on the number of weights and FLOPs that cannot be achieved before.