On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis

Understand

We study the approximation properties and optimization dynamics of recurrent neural networks (RNNs) when applied to learn input-output relationships in temporal data.

• We consider the simple but representative setting of using continuous-time linear RNNs to learn from data generated by linear relationships.
• Mathematically, the latter can be understood as a sequence of linear functionals.
• We prove a universal approximation theorem of such linear functionals, and characterize the approximation rate and its relation with memory.