h-detach: Modifying the LSTM Gradient Towards Better Optimization

Understand

Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP).

• This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately over a large number of steps.
• We introduce a simple stochastic algorithm (\textit{h}-detach) that is specific to LSTM optimization and targeted towards addressing this problem.
• Specifically, we show that when the LSTM weights are large, the gradient components through the linear path (cell state) in the LSTM computational graph get suppressed.