meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting

Understand

We propose a simple yet effective technique for neural network learning.

• The forward propagation is computed as usual.
• In back propagation, only a small subset of the full gradient is computed to update the model parameters.
• The gradient vectors are sparsified in such a way that only the top-$k$ elements (in terms of magnitude) are kept.