2014

Explorations on high dimensional landscapes

Sagun, Levent, Guney, V. Ugur, Arous, Gerard Ben et al.

Understand

Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science.

We provide evidence that some such functions that are defined on high dimensional domains have a narrow band of values whose pre-image contains the bulk of its critical points.
This is in contrast with the low dimensional picture in which this band is wide.
Our simulations agree with the previous theoretical work on spin glasses that proves the existence of such a band when the dimension of the domain tends to infinity.

Dynamics of On-Line Gradient Descent Learning for Multilayer Neural Networks
Saad, David, Birmingham, B, and Solla, Sara A · 1996
Earlier work this paper cites.
The Learning Dynamcis of a Universal Approximator
Original
West, Ansgar H L, Saad, David, and Nabney, Ian T · 1997
Earlier work this paper cites.
Extreme value statistics of eigenvalues of Gaussian random matrices
Dean, David S and Majumdar, Satya N · 2008
Earlier work this paper cites.
A new approach to the planted clique problem
Frieze, Alan and Kannan, Ravi · 2008
Earlier work this paper cites.
Theano: a CPU and GPU math expression compiler
Bergstra, James, Breuleux, Olivier, Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua · 2010
Earlier work this paper cites.
Torch7: A matlab-like environment for machine learning
Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clément · 2011
Earlier work this paper cites.

How glassy are neural networks?
Barra, Adriano, Genovese, Giuseppe, Guerra, Francesco, and Tantari, Daniele · 2012
Cited alongside, same era.
Theano: new features and speed improvements
Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian J., Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua · 2012
Cited alongside, same era.
Complexity of random smooth functions on the high-dimensional sphere
Auffinger, Antonio and Ben Arous, Gérard · 2013
Cited alongside, same era.
Random Matrices and Complexity of Spin Glasses
Auffinger, Antonio, Ben Arous, Gérard, and Černý, Jir̆í · 2013
Cited alongside, same era.
High-Dimensional Random Fields and Random Matrix Theory
Fyodorov, Yan V · 2013
Cited alongside, same era.
Topology Trivialization and Large Deviations for the Minimum in the Simplest Random Optimization
Fyodorov, Yan V and Le Doussal, Pierre · 2013
Cited alongside, same era.
Dark Knowledge
Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff
Cited in the paper.
Energy Landscape of the Finite-Size Mean-field 2-Spin Spherical Model and Topology Trivialization
Mehta, Dhagash, Hauenstein, Jonathan D, Niemerg, Matthew, Simm, Nicholas J, and Stariolo, Daniel A
Cited in the paper.
Energy Landscape of the Finite-Size Mean-field 3-Spin Spherical Model
Mehta, Dhagash, Stariolo, Daniel A, and Kastner, Michael
Cited in the paper.

A walk in the statistical mechanical formulation of neural networks
Agliari, Elena, Barra, Adriano, Galluzzi, Andrea, Tantari, Daniele, and Tavani, Flavia · 2014
Closest in time.
Free Energy and Complexity of Spherical Bipartite Models
Auffinger, Antonio and Chen, Wei-kuo · 2014
Closest in time.
The Loss Surface of Multilayer Networks
Choromanska, Anna, Henaff, Mikael, Mathieu, Michael, Ben Arous, Gérard, and LeCun, Yann · 2014
Closest in time.
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Yann N, Pascanu, Razvan, Gulcehre, Caglar, Cho, Kyunghyun, Ganguli, Surya, and Bengio, Yoshua · 2014
Closest in time.
Physics and Complexity: An Introduction
Sherrington, David · 2014
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…