Rigging the Lottery: Making All Tickets Winners

Understand

Many applications require sparse neural networks due to space or inference time restrictions.

• There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model.
• In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-to-sparse training methods.
• Our method updates the topology of the sparse network during training by using parameter magnitudes and infrequent gradient calculations.