Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory

Understand

The effectiveness of recurrent neural networks can be largely influenced by their ability to store into their dynamical memory information extracted from input sequences at different frequencies and timescales.

• Such a feature can be introduced into a neural architecture by an appropriate modularization of the dynamic memory.
• In this paper we propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning.
• First, we show how to extend the architecture of a simple RNN by separating its hidden state into different modules, each subsampling the network hidden activations at different frequencies.