This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded.
Reading the bibliography…
Mathematical analysis
Apostol, T. M · 1974
Earlier work this paper cites.
The set of all mxn rectangular real matrices of rank-r is connected by analytic regular arcs
Evard, J. C. and Jafari, F · 1994
Earlier work this paper cites.
Learning polynomials with neural networks
Andoni, A., Panigrahy, R., Valiant, G., and Zhang, L · 2014
Earlier work this paper cites.
On the computational efficiency of training neural networks
Livni, R., Shalev-Shwartz, S., and Shamir, O · 2014
Earlier work this paper cites.
The loss surfaces of multilayer networks
Choromanska, A., Hena, M., Mathieu, M., Arous, G. B., and LeCun, Y · 2015
Earlier work this paper cites.
Provable methods for training neural networks with sparse connectivity
Sedghi, H. and Anandkumar, A · 2015
Earlier work this paper cites.
Fast and accurate deep network learning by exponential linear units (elus)
Clevert, D., Unterthiner, T., and Hochreiter, S · 2016
Earlier work this paper cites.
The power of depth for feedforward neural networks
Eldan, R. and Shamir, O · 2016
Earlier work this paper cites.
Globally optimal training of generalized polynomial neural networks with nonlinear spectral methods
Gautier, A., Nguyen, Q., and Hein, M · 2016
Earlier work this paper cites.
Beating the perils of non-convexity: Guaranteed training of neural networks using tensor methods
Janzamin, M., Sedghi, H., and Anandkumar, A · 2016
Earlier work this paper cites.
On the quality of the initial basin in overspecified networks
Safran, I. and Shamir, O · 2016
Earlier work this paper cites.
Benefits of depth in neural networks
Telgarsky, M · 2016
Earlier work this paper cites.
Globally optimal gradient descent for a convnet with gaussian inputs
Brutzkus, A. and Globerson, A · 2017
Earlier work this paper cites.
Global optimality in neural network training
Haeffele, B. D. and Vidal, R · 2017
Earlier work this paper cites.
Identity matters in deep learning
Hardt, M. and Ma, T · 2017
Cited alongside, same era.
Depth creates no bad local minima
Lu, H. and Kawaguchi, K · 2017
Cited alongside, same era.
The loss surface of deep and wide neural networks
Nguyen, Q. and Hein, M · 2017
Cited alongside, same era.
Geometry of neural network loss surfaces via random matrix theory
Pennington, J. and Bahri, Y · 2017
Cited alongside, same era.
Learning relus via gradient descent
Soltanolkotabi, M · 2017
Cited alongside, same era.
Exponentially vanishing sub-optimal local minima in multilayer neural networks
Soudry, D. and Hoffer, E · 2017
Cited alongside, same era.
Diverse neural network learns true target functions
Xie, B., Liang, Y., and Song, L · 2017
Cited alongside, same era.
Global optimality conditions for deep neural networks
Yun, C., Sra, S., and Jadbabaie, A · 2017
Cited alongside, same era.
On the optimization of deep networks: Implicit acceleration by overparameterization
Arora, S., Cohen, N., and Hazan, E · 2018
Cited alongside, same era.
Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks
Bartlett, P., Helmbold, D., and Long, P · 2018
Cited alongside, same era.
On the global convergence of gradient descent for over-parameterized models using optimal transport
Chizat, L. and Bach, F · 2018
Cited alongside, same era.
Essentially no barriers in neural network energy landscape
Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F · 2018
Cited alongside, same era.
Deep linear networks with arbitrary loss: All local minima are global
Laurent, T. and v. Brecht, J · 2018
Later among the works it cites.
Optimization landscape and expressivity of deep cnns
Nguyen, Q. and Hein, M · 2018
Later among the works it cites.
Learning deep models: Critical points and local openness
Nouiehed, M. and Razaviyayn, M · 2018
Later among the works it cites.
An analytical formula of population gradient for two-layered relu network and its applications in convergence and critical point analysis
Tian, Y · 2018
Later among the works it cites.
Spurious valleys in two-layer neural network optimization landscapes
Venturi, L., Bandeira, A. S., and Bruna, J · 2018
Later among the works it cites.
Learning relu networks on linearly separable data: Algorithm, optimality, and generalization
Wang, G., Giannakis, G. B., and Chen, J · 2018
Later among the works it cites.
Deep neural networks with multi-branch architectures are less non-convex
Zhang, H., Shao, J., and Salakhutdinov, R · 2018
Later among the works it cites.
Critical points of neural networks: Analytical forms and landscape properties
Zhou, Y. and Liang, Y · 2018
Later among the works it cites.
Stochastic gradient descent optimizes over-parameterized deep relu networks
Zou, D., Cao, Y., Zhou, D., and Gu, Q · 2018
Later among the works it cites.
A convergence analysis of gradient descent for deep linear neural networks
Arora, S., Cohen, N., Golowich, N., and Hu, W · 2019
Closest in time.
Gradient descent aligns the layers of deep linear networks
Ji, Z. and Telgarsky, M · 2019
Closest in time.
On the loss landscape of a class of deep neural networks with no bad local valleys
Nguyen, Q., Mukkamala, M. C., and Hein, M · 2019
Closest in time.
Eliminating all bad local minima from loss landscapes without even adding an extra unit
Sohl-Dickstein, J. and Kawaguchi, K · 2019
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…