Activation Adaptation in Neural Networks

Understand

Many neural network architectures rely on the choice of the activation function for each hidden layer.

• Given the activation function, the neural network is trained over the bias and the weight parameters.
• The bias catches the center of the activation, and the weights capture the scale.
• Here we propose to train the network over a shape parameter as well.