NoiseOut: A Simple Way to Prune Neural Networks

Understand

Neural networks are usually over-parameterized with significant redundancy in the number of required neurons which results in unnecessary computation and memory usage at inference time.

• One common approach to address this issue is to prune these big networks by removing extra neurons and parameters while maintaining the accuracy.
• In this paper, we propose NoiseOut, a fully automated pruning algorithm based on the correlation between activations of neurons in the hidden layers.
• We prove that adding additional output neurons with entirely random targets results into a higher correlation between neurons which makes pruning by NoiseOut even more efficient.