On the Nonlinearity of Layer Normalization

Understand

Layer normalization (LN) is a ubiquitous technique in deep learning but our theoretical understanding to it remains elusive.

• This paper investigates a new theoretical direction for LN, regarding to its nonlinearity and representation capacity.
• We investigate the representation capacity of a network with layerwise composition of linear and LN transformations, referred to as LN-Net.
• We theoretically show that, given $m$ samples with any label assignment, an LN-Net with only 3 neurons in each layer and $O(m)$ LN layers can correctly classify them.