Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout

Understand

The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights.

• However, these multiple updates can impede optimal training by pulling the model in conflicting directions.
• We present Gradient Sign Dropout (GradDrop), a probabilistic masking procedure which samples gradients at an activation layer based on their level of consistency.
• GradDrop is implemented as a simple deep layer that can be used in any deep net and synergizes with other gradient balancing approaches.