Fetching the paper…

Reading the bibliography…

2016

Revise Saturated Activation Functions

Xu, Bing, Huang, Ruitong, Li, Mu

Original
PDF
Post on X

Understand

In this paper, we revise two commonly used saturated functions, the logistic sigmoid and the hyperbolic tangent (tanh).

We point out that, besides the well-known non-zero centered property, slope of the activation function near the origin is another possible reason making training deep networks with the logistic function difficult to train.
We demonstrate that, with proper rescaling, the logistic sigmoid achieves comparable results with tanh.
Then following the same argument, we improve tahn by penalizing in the negative part.

Revise Saturated Activation Functions · Around

Built on

Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems
Original
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang · 2015
Earlier work this paper cites.

Similar

Fast and accurate deep network learning by exponential linear units (elus)
Original
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Sergey Ioffe and Christian Szegedy · 2015
Cited alongside, same era.

Then

All you need is a good init
Original
Dmytro Mishkin and Jiri Matas · 2015
Later among the works it cites.
Empirical evaluation of rectified activations in convolutional network
Original
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li · 2015
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…