NTK-SAP: Improving neural network pruning by aligning training dynamics

Understand

Pruning neural networks before training has received increasing interest due to its potential to reduce training time and memory.

• One popular method is to prune the connections based on a certain metric, but it is not entirely clear what metric is the best choice.
• Recent advances in neural tangent kernel (NTK) theory suggest that the training dynamics of large enough neural networks is closely related to the spectrum of the NTK.
• Motivated by this finding, we propose to prune the connections that have the least influence on the spectrum of the NTK.