Memory Augmented Neural Networks with Wormhole Connections

Understand

Recent empirical results on long-term dependency tasks have shown that neural networks augmented with an external memory can learn the long-term dependency tasks more easily and achieve better generalization than vanilla recurrent neural networks (RNN).

• We suggest that memory augmented neural networks can reduce the effects of vanishing gradients by creating shortcut (or wormhole) connections.
• Based on this observation, we propose a novel memory augmented neural network model called TARDIS (Temporal Automatic Relation Discovery in Sequences).
• The controller of TARDIS can store a selective set of embeddings of its own previous hidden states into an external memory and revisit them as and when needed.