Built on

Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
Original
Samet Oymak and Mahdi Soltanolkotabi · 1902
Earlier work this paper cites.
Random features for large-scale kernel machines
Ali Rahimi and Benjamin Recht · 2008
Earlier work this paper cites.
Optimistic Rates for Learning with a Smooth Loss
Original
Nathan Srebro, Karthik Sridharan, and Ambuj Tewari · 2010
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Original
Andrew M. Saxe, James L. McClelland, and Surya Ganguli · 2013
Earlier work this paper cites.
On the computational efficiency of training neural networks
Roi Livni, Shai Shalev-Shwartz, and Ohad Shamir · 2014
Earlier work this paper cites.
Understanding machine learning: From theory to algorithms
Shai Shalev-Shwartz and Shai Ben-David · 2014
Earlier work this paper cites.
Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity
Amit Daniely, Roy Frostig, and Yoram Singer · 2016
Earlier work this paper cites.
Deep learning without poor local minima
Kenji Kawaguchi · 2016
Earlier work this paper cites.
Diverse neural network learns true target functions
Original
Bo Xie, Yingyu Liang, and Le Song · 2016
Earlier work this paper cites.
Sgd learns over-parameterized networks that provably generalize on linearly separable data
Original
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz · 2017
Earlier work this paper cites.

Similar

Sgd learns the conjugate kernel class of the network
Amit Daniely · 2017
Cited alongside, same era.
Depth creates no bad local minima
Original
Haihao Lu and Kenji Kawaguchi · 2017
Cited alongside, same era.
Failures of gradient-based deep learning
Shai Shalev-Shwartz, Ohad Shamir, and Shaked Shammah · 2017
Cited alongside, same era.
Learning and generalization in overparameterized neural networks, going beyond two layers
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang · 2018
Cited alongside, same era.
A convergence theory for deep learning via over-parameterization
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song · 2018
Cited alongside, same era.

Then

On the connection between learning two-layers neural networks and tensor decomposition
Original
Marco Mondelli and Andrea Montanari · 2018
Later among the works it cites.
Overparameterized nonlinear learning: Gradient descent takes the shortest path?
Original
Samet Oymak and Mahdi Soltanolkotabi · 2018
Later among the works it cites.
Distribution-specific hardness of learning neural networks
Ohad Shamir · 2018
Later among the works it cites.
Matrix Chernoff bound, November 2018
Wikipedia · 2018
Later among the works it cites.
The Global Optimization Geometry of Shallow Linear Neural Networks
Original
Zhihui Zhu, Daniel Soudry, Yonina C. Eldar, and Michael B. Wakin · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2019

Decoupling Gating from Linearity

Fiat, Jonathan, Malach, Eran, Shalev-Shwartz, Shai

Understand

ReLU neural-networks have been in the focus of many recent theoretical works, trying to explain their empirical success.

Nonetheless, there is still a gap between current theoretical results and empirical observations, even in the case of shallow (one hidden-layer) networks.
For example, in the task of memorizing a random sample of size $m$ and dimension $d$, the best theoretical result requires the size of the network to be $\tilde{\Omega}(\frac{m^2}{d})$, while empirically a network of size slightly larger than $\frac{m}{d}$ is sufficient.
To bridge this gap, we turn to study a simplified model for ReLU networks.

Built on

Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
Original
Samet Oymak and Mahdi Soltanolkotabi · 1902
Earlier work this paper cites.
Random features for large-scale kernel machines
Ali Rahimi and Benjamin Recht · 2008
Earlier work this paper cites.
Optimistic Rates for Learning with a Smooth Loss
Original
Nathan Srebro, Karthik Sridharan, and Ambuj Tewari · 2010
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Original
Andrew M. Saxe, James L. McClelland, and Surya Ganguli · 2013
Earlier work this paper cites.
On the computational efficiency of training neural networks
Roi Livni, Shai Shalev-Shwartz, and Ohad Shamir · 2014
Earlier work this paper cites.
Understanding machine learning: From theory to algorithms
Shai Shalev-Shwartz and Shai Ben-David · 2014
Earlier work this paper cites.
Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity
Amit Daniely, Roy Frostig, and Yoram Singer · 2016
Earlier work this paper cites.
Deep learning without poor local minima
Kenji Kawaguchi · 2016
Earlier work this paper cites.
Diverse neural network learns true target functions
Original
Bo Xie, Yingyu Liang, and Le Song · 2016
Earlier work this paper cites.
Sgd learns over-parameterized networks that provably generalize on linearly separable data
Original
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz · 2017
Earlier work this paper cites.

Similar

Sgd learns the conjugate kernel class of the network
Amit Daniely · 2017
Cited alongside, same era.
Depth creates no bad local minima
Original
Haihao Lu and Kenji Kawaguchi · 2017
Cited alongside, same era.
Failures of gradient-based deep learning
Shai Shalev-Shwartz, Ohad Shamir, and Shaked Shammah · 2017
Cited alongside, same era.
Learning and generalization in overparameterized neural networks, going beyond two layers
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang · 2018
Cited alongside, same era.
A convergence theory for deep learning via over-parameterization
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song · 2018
Cited alongside, same era.

Then

On the connection between learning two-layers neural networks and tensor decomposition
Original
Marco Mondelli and Andrea Montanari · 2018
Later among the works it cites.
Overparameterized nonlinear learning: Gradient descent takes the shortest path?
Original
Samet Oymak and Mahdi Soltanolkotabi · 2018
Later among the works it cites.
Distribution-specific hardness of learning neural networks
Ohad Shamir · 2018
Later among the works it cites.
Matrix Chernoff bound, November 2018
Wikipedia · 2018
Later among the works it cites.
The Global Optimization Geometry of Shallow Linear Neural Networks
Original
Zhihui Zhu, Daniel Soudry, Yonina C. Eldar, and Michael B. Wakin · 2018
Later among the works it cites.