Fetching the paper…

Reading the bibliography…

2020

Neural Collapse with Cross-Entropy Loss

Lu, Jianfeng, Steinerberger, Stefan

Original
PDF
Post on X

Understand

We consider the variational problem of cross-entropy loss with $n$ feature vectors on a unit hypersphere in $\mathbb{R}^d$.

We prove that when $d \geq n - 1$, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior.
We also prove that as $n \rightarrow \infty$ with fixed $d$, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.

Neural Collapse with Cross-Entropy Loss · Around

Built on

D. Bilyk, A. Glazyrin, R. Matzke, J. Park and O. Vlasiuk, Energy on spheres and discreteness of minimizing measures, arXiv:1908.10354,
Original
1908
Earlier work this paper cites.
L. Hormander, The spectral function of an elliptic operator, Acta Math. 121, pp. 193–218 (1968)
1968
Earlier work this paper cites.
J. Benedetto and M. Fickus, Finite Normalized Tight Frames, Advances in Computational Mathematics 18 (2003), 357–385
2003
Earlier work this paper cites.
S. Chopra, R. Hadsell, and Y. LeCun, Learning a similarity metric discriminatively, with application to face verification, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005, pp. 539–546
2005
Earlier work this paper cites.

Similar

A. Bondarenko, D. Radchenko and M. Viazovska, Optimal asymptotic bounds for spherical designs, Annals of Mathematics 178 (2013), p. 443–452
2013
Cited alongside, same era.
T. Mikolov, K. Chen, G. Corrado, and J. Dean, Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781 (2013)
Original
2013
Cited alongside, same era.
W. E and S. Wojtowytsch, On the emergence of tetrahedral symmetry in the final and penultimate layers of neural network classifiers, preprint, arXiv:2012.05420 (2020)
Original
2020
Cited alongside, same era.
A. Jaffe, Y. Kluger, O. Lindenbaum, J. Patsenker, E. Peterfreund and S. Steinerberger, The Spectral Underpinning of word2vec, Front. Appl. Math. Stat., to appear
Cited in the paper.

Then

D. Mixon, H. Parshall, and J. Pi, Neural collapse with unconstrained features, preprint, arXiv:2011.11619 (2020)
Original
2020
Closest in time.
V. Papyan, X. Y. Han and D. Donoho, Prevalence of neural collapse during the terminal phase of deep learning training, Proceedings of the National Academy of Sciences 117 (2020), no. 40, 24652–24663
2020
Closest in time.
T. Wang and P. Isola, Understanding contrastive representation learning through alignment and uniformity on the hypersphere, preprint, arXiv:2005.10242 (2020)
Original
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…