Built on

Compressed sensing
Donoho, David L. 2006 · 2006
Earlier work this paper cites.
Introduction to the non-asymptotic analysis of random matrices
Original
Vershynin, Roman. 2010 · 2010
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
Duchi, John, Hazan, Elad, & Singer, Yoram. 2011 · 2011
Earlier work this paper cites.
Recovery of sparse translation-invariant signals with continuous basis pursuit
Ekanadham, Chaitanya, Tranchina, Daniel, & Simoncelli, Eero P. 2011 · 2011
Earlier work this paper cites.
Lecture 6a Overview of mini–batch gradient descent
Hinton, Geoffrey, Srivastava, N, & Swersky, Kevin. 2012 · 2012
Earlier work this paper cites.
Convex relaxations of structured matrix factorizations
Original
Bach, Francis. 2013 · 2013
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Original
Saxe, Andrew M, McClelland, James L, & Ganguli, Surya. 2013 · 2013
Earlier work this paper cites.

Similar

Compressed sensing off the grid
Tang, Gongguo, Bhaskar, Badri Narayan, Shah, Parikshit, & Recht, Benjamin. 2013 · 2013
Cited alongside, same era.
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Yann N, Pascanu, Razvan, Gulcehre, Caglar, Cho, Kyunghyun, Ganguli, Surya, & Bengio, Yoshua. 2014 · 2014
Cited alongside, same era.
Qualitatively characterizing neural network optimization problems
Original
Goodfellow, Ian J, Vinyals, Oriol, & Saxe, Andrew M. 2014 · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Kingma, Diederik, & Ba, Jimmy. 2014 · 2014
Cited alongside, same era.
Explorations on high dimensional landscapes
Original
Sagun, Levent, Guney, V Ugur, Arous, Gerard Ben, & LeCun, Yann. 2014 · 2014

Then

On the quality of the initial basin in overspecified neural networks
Original
Safran, Itay, & Shamir, Ohad. 2015 · 2015
Later among the works it cites.
Deep Learning without Poor Local Minima
Original
Kawaguchi, Kenji. 2016 · 2016
Closest in time.
Gradient descent converges to minimizers
Lee, Jason D, Simchowitz, Max, Jordan, Michael I, & Recht, Benjamin. 2016 · 2016
Closest in time.
Distribution-Specific Hardness of Learning Neural Networks
Original
Shamir, Ohad. 2016 · 2016
Closest in time.
No bad local minima: Data independent training error guarantees for multilayer neural networks
Original
Soudry, Daniel, & Carmon, Yair. 2016 · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2016

Topology and Geometry of Half-Rectified Network Optimization

Freeman, C. Daniel, Bruna, Joan

Understand

The loss surface of deep neural networks has recently attracted interest in the optimization and machine learning communities as a prime example of high-dimensional non-convex problem.

Some insights were recently gained using spin glass models and mean-field approximations, but at the expense of strongly simplifying the nonlinear nature of the model.
In this work, we do not make any such assumption and study conditions on the data distribution and model architecture that prevent the existence of bad local minima.
Our theoretical work quantifies and formalizes two important \emph{folklore} facts: (i) the landscape of deep linear networks has a radically different topology from that of deep half-rectified ones, and (ii) that the energy landscape in the non-linear case is fundamentally controlled by the interplay between the smoothness of the data distribution and model over-parametrization.

Built on

Compressed sensing
Donoho, David L. 2006 · 2006
Earlier work this paper cites.
Introduction to the non-asymptotic analysis of random matrices
Original
Vershynin, Roman. 2010 · 2010
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
Duchi, John, Hazan, Elad, & Singer, Yoram. 2011 · 2011
Earlier work this paper cites.
Recovery of sparse translation-invariant signals with continuous basis pursuit
Ekanadham, Chaitanya, Tranchina, Daniel, & Simoncelli, Eero P. 2011 · 2011
Earlier work this paper cites.
Lecture 6a Overview of mini–batch gradient descent
Hinton, Geoffrey, Srivastava, N, & Swersky, Kevin. 2012 · 2012
Earlier work this paper cites.
Convex relaxations of structured matrix factorizations
Original
Bach, Francis. 2013 · 2013
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Original
Saxe, Andrew M, McClelland, James L, & Ganguli, Surya. 2013 · 2013
Earlier work this paper cites.

Similar

Compressed sensing off the grid
Tang, Gongguo, Bhaskar, Badri Narayan, Shah, Parikshit, & Recht, Benjamin. 2013 · 2013
Cited alongside, same era.
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Yann N, Pascanu, Razvan, Gulcehre, Caglar, Cho, Kyunghyun, Ganguli, Surya, & Bengio, Yoshua. 2014 · 2014
Cited alongside, same era.
Qualitatively characterizing neural network optimization problems
Original
Goodfellow, Ian J, Vinyals, Oriol, & Saxe, Andrew M. 2014 · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Kingma, Diederik, & Ba, Jimmy. 2014 · 2014
Cited alongside, same era.
Explorations on high dimensional landscapes
Original
Sagun, Levent, Guney, V Ugur, Arous, Gerard Ben, & LeCun, Yann. 2014 · 2014