Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey

Understand

This is a tutorial paper on Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), and their variants.

• We start with a dynamical system and backpropagation through time for RNN.
• Then, we discuss the problems of gradient vanishing and explosion in long-term dependencies.
• We explain close-to-identity weight matrix, long delays, leaky units, and echo state networks for solving this problem.