Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representations

Understand

Temporal models based on recurrent neural networks have proven to be quite powerful in a wide variety of applications.

• However, training these models often relies on back-propagation through time, which entails unfolding the network over many time steps, making the process of conducting credit assignment considerably more challenging.
• Furthermore, the nature of back-propagation itself does not permit the use of non-differentiable activation functions and is inherently sequential, making parallelization of the underlying training process difficult.
• Here, we propose the Parallel Temporal Neural Coding Network (P-TNCN), a biologically inspired model trained by the learning algorithm we call Local Representation Alignment.