Fetching the paper…

Reading the bibliography…

2024

Geometry and Dynamics of LayerNorm

Riechers, Paul M.

Original
PDF
Post on X

Understand

A technical note aiming to offer deeper intuition for the LayerNorm function common in deep neural networks.

LayerNorm is defined relative to a distinguished 'neural' basis, but it does more than just normalize the corresponding vector elements.
Rather, it implements a composition -- of linear projection, nonlinear scaling, and then affine transformation -- on input activation vectors.
We develop both a new mathematical expression and geometric intuition, to make the net effect more transparent.

Geometry and Dynamics of LayerNorm · Around

Built on

Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Earlier work this paper cites.
Beyond the spectral theorem: Decomposing arbitrary functions of nondiagonalizable operators
P. M. Riechers and J. P. Crutchfield · 2018
Earlier work this paper cites.

Similar

On layer normalization in the transformer architecture
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu · 2020
Cited alongside, same era.
On the expressivity role of LayerNorm in transformers’ attention
Original
Shaked Brody, Uri Alon, and Eran Yahav · 2023
Cited alongside, same era.

Then

Traveling words: A geometric interpretation of transformers
Original
Raul Molina · 2023
Later among the works it cites.
https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html
LayerNorm — PyTorch 2.3 documentation · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…