Quantifying the vanishing gradient and long distance dependency problem in recursive neural networks and recursive LSTMs

Understand

Recursive neural networks (RNN) and their recently proposed extension recursive long short term memory networks (RLSTM) are models that compute representations for sentences, by recursively combining word embeddings according to an externally provided parse tree.

• Both models thus, unlike recurrent networks, explicitly make use of the hierarchical structure of a sentence.
• In this paper, we demonstrate that RNNs nevertheless suffer from the vanishing gradient and long distance dependency problem, and that RLSTMs greatly improve over RNN's on these problems.
• We present an artificial learning task that allows us to quantify the severity of these problems for both models.