Stochastic Gradient Descent Learns State Equations with Nonlinear Activations

Understand

We study discrete time dynamical systems governed by the state equation $h_{t+1}=\phi(Ah_t+Bu_t)$.

• Here $A,B$ are weight matrices, $\phi$ is an activation function, and $u_t$ is the input data.
• This relation is the backbone of recurrent neural networks (e.g.
• LSTMs) which have broad applications in sequential learning tasks.