Soft Threshold Weight Reparameterization for Learnable Sparsity

Understand

Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus of maximizing prediction accuracy given an overall parameter budget.

• Existing methods rely on uniform or heuristic non-uniform sparsity budgets which have sub-optimal layer-wise parameter allocation resulting in a) lower prediction accuracy or b) higher inference cost (FLOPs).
• This work proposes Soft Threshold Reparameterization (STR), a novel use of the soft-threshold operator on DNN weights.
• STR smoothly induces sparsity while learning pruning thresholds thereby obtaining a non-uniform sparsity budget.