Deep Learning with S-shaped Rectified Linear Activation Units

Understand

Rectified linear activation units are important components for state-of-the-art deep convolutional networks.

• In this paper, we propose a novel S-shaped rectified linear activation unit (SReLU) to learn both convex and non-convex functions, imitating the multiple function forms given by the two fundamental laws, namely the Webner-Fechner law and the Stevens law, in psychophysics and neural sciences.
• Specifically, SReLU consists of three piecewise linear functions, which are formulated by four learnable parameters.
• The SReLU is learned jointly with the training of the whole deep network through back propagation.