Sparse maximal update parameterization: A holistic approach to sparse training dynamics

Understand

Several challenges make it difficult for sparse neural networks to compete with dense models.

• First, setting a large fraction of weights to zero impairs forward and gradient signal propagation.
• Second, sparse studies often need to test multiple sparsity levels, while also introducing new hyperparameters (HPs), leading to prohibitive tuning costs.
• Indeed, the standard practice is to re-use the learning HPs originally crafted for dense models.