Fetching the paper…

Reading the bibliography…

Survey of Expressivity in Deep Neural Networks · Around

2016

Survey of Expressivity in Deep Neural Networks

Raghu, Maithra, Poole, Ben, Kleinberg, Jon et al.

Original
PDF
Post on X

Understand

We survey results on neural network expressivity described in "On the Expressive Power of Deep Neural Networks".

The paper motivates and develops three natural measures of expressiveness, which all display an exponential dependence on the depth of the network.
In fact, all of these measures are related to a fourth quantity, trajectory length.
This quantity grows exponentially in the depth of the network, and is responsible for the depth sensitivity observed.

Built on

Multilayer feedforward networks are universal approximators
Kurt Hornik, Maxwell Stinchcombe, and Halbert White · 1989
Earlier work this paper cites.
Approximation by superpositions of a sigmoidal function
George Cybenko · 1989
Earlier work this paper cites.
On the representational efficiency of restricted boltzmann machines
James Martens, Arkadev Chattopadhya, Toni Pitassi, and Richard Zemel · 2013
Earlier work this paper cites.

Similar

On the number of response regions of deep feed forward networks with piece-wise linear activations
Original
Razvan Pascanu, Guido Montufar, and Yoshua Bengio · 2013
Cited alongside, same era.
On the complexity of neural network classifiers: A comparison between shallow and deep architectures
Monica Bianchini and Franco Scarselli · 2014
Cited alongside, same era.
The power of depth for feedforward neural networks
Original
Ronen Eldan and Ohad Shamir · 2015
Cited alongside, same era.

Then

Representation benefits of deep feedforward networks
Original
Matus Telgarsky · 2015
Later among the works it cites.
On the expressive power of deep neural networks
Original
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein · 2016
Closest in time.
Exponential expressivity in deep neural networks through transient chaos
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…