Fetching the paper…

Reading the bibliography…

2018

Fast Weight Long Short-Term Memory

Keller, T. Anderson, Sridhar, Sharath Nittur, Wang, Xin

Original
PDF
Post on X

Understand

Associative memory using fast weights is a short-term memory mechanism that substantially improves the memory capacity and time scale of recurrent neural networks (RNNs).

As recent studies introduced fast weights only to regular RNNs, it is unknown whether fast weight memory is beneficial to gated RNNs.
In this work, we report a significant synergy between long short-term memory (LSTM) networks and fast weight associative memories.
We show that this combination, in learning associative retrieval tasks, results in much faster training and lower test error, a performance boost most prominent at high memory task difficulties.

Fast Weight Long Short-Term Memory · Around

Built on

Recurrent Highway Networks
Original
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber · 1938
Earlier work this paper cites.
Using Fast Weights to Deblur Old Memories
Geoffrey E. Hinton and David C. Plaut · 1987
Earlier work this paper cites.
Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks
Jürgen Schmidhuber · 1992
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jurgen Schmidhuber · 1997
Earlier work this paper cites.
Neural Turing Machines
Original
Alex Graves, Greg Wayne, and Ivo Danihelka · 2007
Earlier work this paper cites.
Neural Machine Translation by Jointly Learning to Align and Translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2014
Earlier work this paper cites.

Similar

Memory networks
Original
Jason Weston, Sumit Chopra, and Antoine Bordes · 2015
Cited alongside, same era.
Hierarchical Multiscale Recurrent Neural Networks
Original
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio · 2016
Cited alongside, same era.
Associative Long Short-Term Memory
Original
Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, and Alex Graves · 2016
Cited alongside, same era.
Rotational Unit of Memory
Original
Rumen Dangovski, Li Jing, and Marin Soljacic · 2017
Cited alongside, same era.
Frustratingly Short Attention Spans in Neural Language Modeling
Original

Then

Learning to Remember Rare Events
Original
Łukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio · 2017
Later among the works it cites.
Nested LSTMs
Original
Joel Ruben, Antony Moniz, David Krueger, and David Krueger@umontreal Ca · 2017
Later among the works it cites.
Gated Fast Weights for On-The-Fly Neural Program Generation
Imanol Schlag and Jürgen Schmidhuber · 2017
Later among the works it cites.
Attention Is All You Need
Original
Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Later among the works it cites.
Learning to update Auto-associative Memory in Recurrent Neural Networks for Improving Sequence Memorization
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Michał Daniluk, Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel · 2017

Cited alongside, same era.

Using Fast Weights to Attend to the Recent Past

Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu

Cited in the paper.

Layer Normalization

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton

Cited in the paper.

Wei Zhang and Bowen Zhou · 2017

Later among the works it cites.