Node Perturbation Can Effectively Train Multi-Layer Neural Networks

Understand

Backpropagation (BP) remains the dominant and most successful method for training parameters of deep neural network models.

• However, BP relies on two computationally distinct phases, does not provide a satisfactory explanation of biological learning, and can be challenging to apply for training of networks with discontinuities or noisy node dynamics.
• By comparison, node perturbation (NP), also known as activity-perturbed forward gradients, proposes learning by the injection of noise into network activations, and subsequent measurement of the induced loss change.
• NP relies on two forward (inference) passes, does not make use of network derivatives, and has been proposed as a model for learning in biological systems.