Learning Neural Activations

Understand

An artificial neuron is modelled as a weighted summation followed by an activation function which determines its output.

• A wide variety of activation functions such as rectified linear units (ReLU), leaky-ReLU, Swish, MISH, etc.
• have been explored in the literature.
• In this short paper, we explore what happens when the activation function of each neuron in an artificial neural network is learned natively from data alone.