Fetching the paper…

Reading the bibliography…

Pure and Spurious Critical Points: a Geometric Study of Linear Networks · Around

2019

Pure and Spurious Critical Points: a Geometric Study of Linear Networks

Trager, Matthew, Kohn, Kathlén, Bruna, Joan

Original
PDF
Post on X

Understand

The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network's weights.

We introduce a natural distinction between pure critical points, which only depend on the functional space, and spurious critical points, which arise from the parameterization.
We apply this perspective to revisit and extend the literature on the loss function of linear neural networks.
For this type of network, the functional space is either the set of all linear maps from input to output space, or a determinantal variety, i.e., a set of linear maps with bounded rank.

Built on

Neural networks and principal component analysis: Learning from examples without local minima
Pierre Baldi and Kurt Hornik · 1989
Earlier work this paper cites.
Algebraic Geometry: A First Course
Joe Harris · 1995
Earlier work this paper cites.
Critical Points of Neural Networks: Analytical Forms and Landscape Properties
Original
Yi Zhou and Yingbin Liang · 1995
Earlier work this paper cites.
Introduction to Smooth Manifolds
John M. Lee · 2003
Earlier work this paper cites.
The Euclidean distance degree of an algebraic variety
Original
Jan Draisma, Emil Horobet, Giorgio Ottaviani, Bernd Sturmfels, and Rekha R. Thomas · 2013
Earlier work this paper cites.
Exact Solutions in Structured Low-Rank Approximation
Original
Giorgio Ottaviani, Pierre-Jean Spaenlehauer, and Bernd Sturmfels · 2013
Earlier work this paper cites.
The average number of critical rank-one approximations to a tensor
Original
Jan Draisma and Emil Horobet · 2014
Earlier work this paper cites.
The Loss Surfaces of Multilayer Networks
Anna Choromanska, Mikael Henaff, Michaël Mathieu, Gérard Ben Arous, and Yann LeCun · 2015
Earlier work this paper cites.
Information Geometry and Its Applications , volume 194 of Applied Mathematical Sciences
Shun-ichi Amari · 2016
Earlier work this paper cites.

Similar

Identity Matters in Deep Learning
Original
Moritz Hardt and Tengyu Ma · 2016
Cited alongside, same era.
Deep Learning without Poor Local Minima
Original
Kenji Kawaguchi · 2016
Cited alongside, same era.
Deep linear neural networks with arbitrary loss: All local minima are global
Original
Thomas Laurent and James von Brecht · 2017
Cited alongside, same era.
Depth Creates No Bad Local Minima
Original
Haihao Lu and Kenji Kawaguchi · 2017
Cited alongside, same era.
Global optimality conditions for deep neural networks
Original
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie · 2017

Then

A Mean Field View of the Landscape of Two-Layers Neural Networks
Original
Song Mei, Andrea Montanari, and Phan-Minh Nguyen · 2018
Later among the works it cites.
Spurious valleys in two-layer neural network optimization landscapes
Original
Luca Venturi, Afonso S Bandeira, and Joan Bruna · 2018
Later among the works it cites.
Small nonlinearities in activation functions create bad local minima in neural networks
Original
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie · 2018
Later among the works it cites.
Implicit regularization in deep matrix factorization
Original
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo · 2019
Closest in time.
Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Cited alongside, same era.

A convergence analysis of gradient descent for deep linear neural networks

Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu · 2018

Cited alongside, same era.

On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport

Lenaic Chizat and Francis Bach · 2018

Cited alongside, same era.

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson · 2018

Cited alongside, same era.

The loss surface of deep linear networks viewed through the algebraic geometry lens

Original

Dhagash Mehta, Tianran Chen, Tingting Tang, and Jonathan D. Hauenstein · 2018

Cited alongside, same era.

Bubacarr Bah, Holger Rauhut, Ulrich Terstiege, and Michael Westdickenberg · 2019

Closest in time.

Macaulay2, a software system for research in algebraic geometry

Daniel R. Grayson and Michael E. Stillman · 2019

Closest in time.

Learning algebraic models of quantum entanglement

Hamza Jaffali and Luke Oeding · 2019

Closest in time.

On the expressive power of deep polynomial neural networks

Joe Kileel, Matthew Trager, and Joan Bruna · 2019

Closest in time.

Depth creates no more spurious local minima

Li Zhang · 2019

Closest in time.