Fetching the paper…

Reading the bibliography…

2019

Symmetrical Gaussian Error Linear Units (SGELUs)

Yu, Chao, Su, Zhiguo

Original
PDF
Post on X

Understand

In this paper, a novel neural network activation function, called Symmetrical Gaussian Error Linear Unit (SGELU), is proposed to obtain high performance.

It is achieved by effectively integrating the property of the stochastic regularizer in the Gaussian Error Linear Unit (GELU) with the symmetrical characteristics.
Combining with these two merits, the proposed unit introduces the capability of the bidirection convergence to successfully optimize the network without the gradient diminishing problem.
The evaluations of SGELU against GELU and Linearly Scaled Hyperbolic Tangent (LiSHT) have been carried out on MNIST classification and MNIST auto-encoder, which provide great validations in terms of the performance, the convergence rate among these applications.

Symmetrical Gaussian Error Linear Units (SGELUs) · Around

Built on

Lisht: Non-parametric linearly scaled hyperbolic tangent activation function for neural networks
Original
Roy, S. K., Manna, S., Dubey, S. R., and Chaudhuri, B. B. (2019) · 1901
Earlier work this paper cites.
Handbook of methods of applied statistics
Brillinger, D. R., Chakravarti, I. M., Laha, R. G., and Roy, J · 1967
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
Nair, V. and Hinton, G. E · 2010
Earlier work this paper cites.

Similar

Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Cited alongside, same era.
Fast and accurate deep network learning by exponential linear units (elus)
Original
Clevert, D.-A., Unterthiner, T., and Hochreiter, S · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Ioffe, S. and Szegedy, C · 2015
Cited alongside, same era.

Then

Gaussian error linear units (gelus)
Original
Hendrycks, D. and Gimpel, K · 2016
Later among the works it cites.
Swish: a self-gated activation function
Original
Ramachandran, P., Zoph, B., and Le, Q. V · 2017
Later among the works it cites.
A general reinforcement learning algorithm that masters chess, shogi, and go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. (2018) · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…