The loss landscape of overparameterized neural networks

Understand

We explore some mathematical features of the loss landscape of overparameterized neural networks.

• A priori one might imagine that the loss function looks like a typical function from $\mathbb{R}^n$ to $\mathbb{R}$ - in particular, nonconvex, with discrete global minima.
• In this paper, we prove that in at least one important way, the loss function of an overparameterized neural network does not look like a typical function.
• If a neural net has $n$ parameters and is trained on $d$ data points, with $n>d$, we show that the locus $M$ of global minima of $L$ is usually not discrete, but rather an $n-d$ dimensional submanifold of $\mathbb{R}^n$.