Loss Landscape Degeneracy and Stagewise Development in Transformers

Understand

Deep learning involves navigating a high-dimensional loss landscape over the neural network parameter space.

• Over the course of training, complex computational structures form and re-form inside the neural network, leading to shifts in input/output behavior.
• It is a priority for the science of deep learning to uncover principles governing the development of neural network structure and behavior.
• Drawing on the framework of singular learning theory, we propose that model development is deeply linked to degeneracy in the local geometry of the loss landscape.