Diminishing Batch Normalization

Understand

In this paper, we propose a generalization of the Batch Normalization (BN) algorithm, diminishing batch normalization (DBN), where we update the BN parameters in a diminishing moving average way.

• BN is very effective in accelerating the convergence of a neural network training phase that it has become a common practice.
• Our proposed DBN algorithm remains the overall structure of the original BN algorithm while introduces a weighted averaging update to some trainable parameters.
• We provide an analysis of the convergence of the DBN algorithm that converges to a stationary point with respect to trainable parameters.