A survey on modern trainable activation functions

Understand

In neural networks literature, there is a strong interest in identifying and defining activation functions which can improve neural network performance.

• In recent years there has been a renovated interest of the scientific community in investigating activation functions which can be trained during the learning process, usually referred to as "trainable", "learnable" or "adaptable" activation functions.
• They appear to lead to better network performance.
• Diverse and heterogeneous models of trainable activation function have been proposed in the literature.