Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks

Understand

Batch Normalization (BN) uses mini-batch statistics to normalize the activations during training, introducing dependence between mini-batch elements.

• This dependency can hurt the performance if the mini-batch size is too small, or if the elements are correlated.
• Several alternatives, such as Batch Renormalization and Group Normalization (GN), have been proposed to address this issue.
• However, they either do not match the performance of BN for large batches, or still exhibit degradation in performance for smaller batches, or introduce artificial constraints on the model architecture.