Gated Orthogonal Recurrent Units: On Learning to Forget

Understand

We present a novel recurrent neural network (RNN) based model that combines the remembering ability of unitary RNNs with the ability of gated RNNs to effectively forget redundant/irrelevant information in its memory.

• We achieve this by extending unitary RNNs with a gating mechanism.
• Our model is able to outperform LSTMs, GRUs and Unitary RNNs on several long-term dependency benchmark tasks.
• We empirically both show the orthogonal/unitary RNNs lack the ability to forget and also the ability of GORU to simultaneously remember long term dependencies while forgetting irrelevant information.