Memory and Information Processing in Recurrent Neural Networks

Understand

Recurrent neural networks (RNN) are simple dynamical systems whose computational power has been attributed to their short-term memory.

• Short-term memory of RNNs has been previously studied analytically only for the case of orthogonal networks, and only under annealed approximation, and uncorrelated input.
• Here for the first time, we present an exact solution to the memory capacity and the task-solving performance as a function of the structure of a given network instance, enabling direct determination of the function--structure relation in RNNs.
• We calculate the memory capacity for arbitrary networks with exponentially correlated input and further related it to the performance of the system on signal processing tasks in a supervised learning setup.