Can recurrent neural networks warp time?

Understand

Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use ad hoc gating mechanisms.

• Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues.
• We prove that learnable gates in a recurrent model formally provide quasi- invariance to general time transformations in the input data.
• We recover part of the LSTM architecture from a simple axiomatic approach.