Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization

Understand

Deep convolutional neural networks are known to be unstable during training at high learning rate unless normalization techniques are employed.

• Normalizing weights or activations allows the use of higher learning rates, resulting in faster convergence and higher test accuracy.
• Batch normalization requires minibatch statistics that approximate the dataset statistics but this incurs additional compute and memory costs and causes a communication bottleneck for distributed training.
• Weight normalization and initialization-only schemes do not achieve comparable test accuracy.