Fetching the paper…

Reading the bibliography…

2015

Feedforward Sequential Memory Neural Networks without Recurrent Feedback

Zhang, ShiLiang, Jiang, Hui, Wei, Si et al.

Original
PDF
Post on X

Understand

We introduce a new structure for memory neural networks, called feedforward sequential memory networks (FSMN), which can learn long-term dependency without using recurrent feedback.

The proposed FSMN is a standard feedforward neural networks equipped with learnable sequential memory blocks in the hidden layers.
In this work, we have applied FSMN to several language modeling (LM) tasks.
Experimental results have shown that the memory blocks in FSMN can learn effective representations of long history.

Feedforward Sequential Memory Neural Networks without Recurrent Feedback · Around

Built on

Y. Bengio, P. Simard, P. Frasconi. Learning long-term dependencies with gradient descent is difficult
1994
Earlier work this paper cites.
S. Hochreiter, J. Schmidhuber. Long short-term memory
1997
Earlier work this paper cites.
G. Xavier, Y. Bengio. Understanding the difficulty of training deep feedforward neural networks
2010
Earlier work this paper cites.
T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ and S. Khudanpur. Recurrent neural network based language model
Original
2010
Earlier work this paper cites.
T. Mikolov, S. Kombrink, L. Burget, J. Cernocky and S. Khudanpur. Extensions of recurrent neural network language model
2011
Earlier work this paper cites.

Similar

M. Mahoney. Large Text Compression Benchmark
2011
Cited alongside, same era.
A. Graves. Generating sequences with recurrent neural networks
Original
2013
Cited alongside, same era.
J. Chung, C. Gulcehre, K. H. Cho, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling
Original
2014
Cited alongside, same era.
T. Mikolov, A. Joulin, S. Chopra, et al. Learning longer memory in recurrent neural networks
Original
2014
Cited alongside, same era.
A. Graves, G. Wayne, I. Danihelka. Neural Turing Machines
Original
2014
Cited alongside, same era.

Then

J. Weston, S. Chopra, A. Bordes. Memory networks
Original
2014
Later among the works it cites.
S. Sukhbaatar, A. Szlam, J. Weston. End-To-End Memory Networks
Original
2015
Closest in time.
A. Joulin, T. Mikolov. Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
Original
2015
Closest in time.
A. Bordes, N. Usunier, S. Chopra, J. Weston. Large-scale Simple Question Answering with Memory Networks
Original
2015
Closest in time.
S. Zhang, H. Jiang, M. Xu, J. Hou and L.R. Dai. The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models
2015

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Closest in time.