Training Neural Networks with Fixed Sparse Masks

Understand

During typical gradient-based training of deep neural networks, all of the model's parameters are updated at each iteration.

• Recent work has shown that it is possible to update only a small subset of the model's parameters during training, which can alleviate storage and communication requirements.
• In this paper, we show that it is possible to induce a fixed sparse mask on the model's parameters that selects a subset to update over many iterations.
• Our method constructs the mask out of the $k$ parameters with the largest Fisher information as a simple approximation as to which parameters are most important for the task at hand.