Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data

Understand

Modern deep neural networks have achieved impressive performance on tasks from image classification to natural language processing.

• Surprisingly, these complex systems with massive amounts of parameters exhibit the same structural properties in their last-layer features and classifiers across canonical datasets when training until convergence.
• In particular, it has been observed that the last-layer features collapse to their class-means, and those class-means are the vertices of a simplex Equiangular Tight Frame (ETF).
• This phenomenon is known as Neural Collapse (NC).