Learning Sparse Neural Networks through $L_0$ Regularization

Understand

We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero.

• Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization.
• AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization.
• However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function.