Fetching the paper…

Reading the bibliography…

2018

Are Efficient Deep Representations Learnable?

Nye, Maxwell, Saxe, Andrew

Original
PDF
Post on X

Understand

Many theories of deep learning have shown that a deep network can require dramatically fewer resources to represent a given function compared to a shallow network.

But a question remains: can these efficient representations be learned using current deep learning techniques? In this work, we test whether standard deep learning methods can in fact find the efficient representations posited by several theories of deep representation.
Specifically, we train deep neural networks to learn two simple functions with known efficient solutions: the parity function and the fast Fourier transform.
We find that using gradient-based optimization, a deep network does not learn the parity function, unless initialized very close to a hand-coded exact solution.

Are Efficient Deep Representations Learnable? · Around

Built on

Scaling learning algorithms towards AI
Yoshua Bengio and Yann LeCun · 2007
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
X. Glorot and Y. Bengio · 2010
Earlier work this paper cites.
On the number of inference regions of deep feed forward networks with piece-wise linear activations
Original
Razvan Pascanu, Guido Montúfar, and Yoshua Bengio · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.

Similar

On the number of linear regions of deep neural networks
G. Montufar, K. Cho, R. Pascanu, and Y. Bengio · 2014
Cited alongside, same era.
Representation Benefits of Deep Feedforward Networks
Original
M. Telgarsky · 2015
Cited alongside, same era.
The Power of Depth for Feedforward Neural Networks
R. Eldan and O. Shamir · 2016
Cited alongside, same era.
Optimal Architectures in a Solvable Model of Deep Networks
J. Kadmon and H. Sompolinsky · 2016
Cited alongside, same era.

Then

Learning Functions: When Is Deep Better Than Shallow
H. Mhaskar, Q. Liao, and T. Poggio · 2016
Later among the works it cites.
Why and when can deep - but not shallow - networks avoid the curse of dimensionality: a review
Original
Tomaso A. Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, and Qianli Liao · 2016
Later among the works it cites.
Exponential expressivity in deep neural networks through transient chaos
B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli · 2016
Later among the works it cites.
Theory of Deep Learning II: Landscape of the Empirical Risk in Deep Learning by
Q. Liao and T. Poggio · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…