Memorization-Dilation: Modeling Neural Collapse Under Label Noise

Understand

The notion of neural collapse refers to several emergent phenomena that have been empirically observed across various canonical classification problems.

• During the terminal phase of training a deep neural network, the feature embedding of all examples of the same class tend to collapse to a single representation, and the features of different classes tend to separate as much as possible.
• Neural collapse is often studied through a simplified model, called the unconstrained feature representation, in which the model is assumed to have "infinite expressivity" and can map each data point to any arbitrary representation.
• In this work, we propose a more realistic variant of the unconstrained feature representation that takes the limited expressivity of the network into account.