Built on

M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken, “Multilayer feedforward networks with a nonpolynomial activation function can approximate any function,” Neural Networks
1993
Earlier work this paper cites.
A. Krizhevsky, “Learning multiple layers of features from tiny images,” tech. rep., 2009
Original
2009
Earlier work this paper cites.

Similar

R. Arora, A. Basu, P. Mianjy, and A. Mukherjee, “Understanding deep neural networks with rectified linear units,” 2016
2016
Cited alongside, same era.
B. Xu, R. Huang, and M. Li, “Revise saturated activation functions,” 2016
2016
Cited alongside, same era.

Then

P. Ramachandran, B. Zoph, and Q. V. Le, “Searching for activation functions,” 2017
2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2018

PLU: The Piecewise Linear Unit Activation Function

Nicolae, Andrei

Understand

Successive linear transforms followed by nonlinear "activation" functions can approximate nonlinear functions to arbitrary precision given sufficient layers.

The number of necessary layers is dependent on, in part, by the nature of the activation function.
The hyperbolic tangent (tanh) has been a favorable choice as an activation until the networks grew deeper and the vanishing gradients posed a hindrance during training.
For this reason the Rectified Linear Unit (ReLU) defined by max(0, x) has become the prevailing activation function in deep neural networks.