Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations

Built on

Approximation by superpositions of a sigmoidal function
G. Cybenko · 1989
Earlier work this paper cites.
Multilayer feedforward networks are universal approximators
K. Hornik, M. Stinchcombe, H. White · 1989
Earlier work this paper cites.
Deep learning
Y. Bengio, G. Hinton, and Y. LeCun · 2015
Earlier work this paper cites.
Near-optimal max-affine estimators for convex regression
G. Balázs, A, György, and C. Szepesvári · 2015
Earlier work this paper cites.
Representation benefits of deep feedforward networks
Original
M. Telgrasky · 2015
Earlier work this paper cites.

Learning functions: when is deep better than shallow
Original
Q. Liao, H. Mhaskar, and T. Poggio · 2016
Cited alongside, same era.
Why does deep and cheap learning work so well?
Original
H. Lin, D. Rolnick, M. Tegmark · 2016
Cited alongside, same era.
Deep vs. shallow networks: an approximation theory perspective
Original
H. Mhaskar, T. Poggio · 2016
Cited alongside, same era.
Exponential expressivity in deep neural networks through transient chaos
Original
B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, S. Ganguli · 2016
Cited alongside, same era.
Benefits of depth in neural nets
M. Telgrasky · 2016
Cited alongside, same era.
Understanding deep neural networks with Rectified Linear Units
Original
R. Arora, A. Basu, P. Mianjy, A. Mukherjee
Cited in the paper.
Expressivity of ReLU \Relu nets on the simplex. In preparation
B. Hanin, E. Mossel
Cited in the paper.
Approximating Continuous Functions by ReLU Nets of Minimal Width. arXiv:1710.11278
Original
B. Hanin, M. Sellke
Cited in the paper.
Mathematical aspects of deep learning: http://elmos.scripts.mit.edu/mathofdeeplearning/mathematical-aspects-of-deep-learning-intro/
E. Mossel
Cited in the paper.