On orthogonality and learning recurrent networks with long term dependencies

Understand

It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies.

• The vanishing or exploding gradient problem is a well known issue associated with these challenges.
• One approach to addressing vanishing and exploding gradients is to use either soft or hard constraints on weight matrices so as to encourage or enforce orthogonality.
• Orthogonal matrices preserve gradient norm during backpropagation and may therefore be a desirable property.