On Feature Normalization and Data Augmentation

Understand

The moments (a.k.a., mean and standard deviation) of latent features are often removed as noise when training image recognition models, to increase stability and reduce training time.

• However, in the field of image generation, the moments play a much more central role.
• Studies have shown that the moments extracted from instance normalization and positional normalization can roughly capture style and shape information of an image.
• Instead of being discarded, these moments are instrumental to the generation process.