Approximate Fixed-Points in Recurrent Neural Networks

Understand

Recurrent neural networks are widely used in speech and language processing.

• Due to dependency on the past, standard algorithms for training these models, such as back-propagation through time (BPTT), cannot be efficiently parallelised.
• Furthermore, applying these models to more complex structures than sequences requires inference time approximations, which introduce inconsistency between inference and training.
• This paper shows that recurrent neural networks can be reformulated as fixed-points of non-linear equation systems.