L2 Regularization versus Batch and Weight Normalization

Understand

Batch Normalization is a commonly used trick to improve the training of deep neural networks.

• These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting.
• However, we show that L2 regularization has no regularizing effect when combined with normalization.
• Instead, regularization has an influence on the scale of weights, and thereby on the effective learning rate.