Long Expressive Memory for Sequence Modeling

Understand

We propose a novel method called Long Expressive Memory (LEM) for learning long-term sequential dependencies.

• LEM is gradient-based, it can efficiently process sequential tasks with very long-term dependencies, and it is sufficiently expressive to be able to learn complicated input-output maps.
• To derive LEM, we consider a system of multiscale ordinary differential equations, as well as a suitable time-discretization of this system.
• For LEM, we derive rigorous bounds to show the mitigation of the exploding and vanishing gradients problem, a well-known challenge for gradient-based recurrent sequential learning methods.