Fetching the paper…

Reading the bibliography…

2016

Stuck in a What? Adventures in Weight Space

Lipton, Zachary C.

Original
PDF
Post on X

Understand

Deep learning researchers commonly suggest that converged models are stuck in local minima.

More recently, some researchers observed that under reasonable assumptions, the vast majority of critical points are saddle points, not true minima.
Both descriptions suggest that weights converge around a point in weight space, be it a local optima or merely a critical point.
However, it's possible that neither interpretation is accurate.

Stuck in a What? Adventures in Weight Space · Around

Built on

The mnist database of handwritten digits, 1998
Original
Yann LeCun, Corinna Cortes, and Christopher JC Burges · 1998
Earlier work this paper cites.
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio · 2014
Earlier work this paper cites.

Similar

Qualitatively characterizing neural network optimization problems
Original
Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe · 2014
Cited alongside, same era.

Then

Beating the perils of non-convexity: Guaranteed training of neural networks using tensor methods
Original
Majid Janzamin, Hanie Sedghi, and Anima Anandkumar · 2015
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…