2015

A Critical Review of Recurrent Neural Networks for Sequence Learning

Lipton, Zachary C., Berkowitz, John, Elkan, Charles

Understand

Countless learning tasks require dealing with sequential data.

Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences.
In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences.
Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities.

Built on

Computing machinery and intelligence
Alan M. Turing · 1950
Earlier work this paper cites.
Conditional markov processes
Ruslan L. Stratonovich · 1960
Earlier work this paper cites.
Error bounds for convolutional codes and an asymptotically optimum decoding algorithm
Andrew J. Viterbi · 1967
Earlier work this paper cites.
Neural networks and physical systems with emergent collective computational abilities
John J. Hopfield · 1982
Earlier work this paper cites.
Learning internal representations by error propagation
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams · 1985
Earlier work this paper cites.
Learning distributed representations of concepts, 1986
Geoffrey E. Hinton · 1986
Earlier work this paper cites.
Serial order: A parallel distributed processing approach
Michael I. Jordan · 1986
Earlier work this paper cites.
A learning algorithm for continually running fully recurrent neural networks
Ronald J. Williams and David Zipser · 1989
Earlier work this paper cites.
Evolving networks: Using the genetic algorithm with connectionist learning
Richard K. Belew, John McInerney, and Nicol N. Schraudolph · 1990
Earlier work this paper cites.
Finding structure in time
Jeffrey L. Elman · 1990
Earlier work this paper cites.
Handwritten digit recognition with a back-propagation network
Yann Le Cun, B. Boser, John S. Denker, D. Henderson, Richard E. Howard, W. Hubbard, and Lawrence D. Jackel · 1990
Earlier work this paper cites.
Backpropagation through time: what it does and how to do it
Paul J. Werbos · 1990
Earlier work this paper cites.
Turing computability with neural nets
Hava T. Siegelmann and Eduardo D. Sontag · 1991
Earlier work this paper cites.
Training a 3-node neural network is NP-complete
Avrim L. Blum and Ronald L. Rivest · 1993
Earlier work this paper cites.
Learning long-term dependencies with gradient descent is difficult
Yoshua Bengio, Patrice Simard, and Paolo Frasconi · 1994
Earlier work this paper cites.
Neural network synthesis using cellular encoding and the genetic algorithm., 1994
Frédéric Gruau, L’universite Claude Bernard lyon I, Of A Diplome De Doctorat, M. Jacques Demongeot, Examinators M. Michel Cosnard, M. Jacques Mazoyer, M. Pierre Peretto, and M. Darell Whitley · 1994
Earlier work this paper cites.
Gradient calculations for dynamic recurrent neural networks: A survey
Barak A. Pearlmutter · 1995
Earlier work this paper cites.
Working memory and executive control [and discussion]
Alan Baddeley, Sergio Della Sala, and T.W. Robbins · 1996
Earlier work this paper cites.
Bridging long time lags by weight guessing and “long short-term memory”
Sepp Hochreiter and Jurgen Schmidhuber · 1996
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Bidirectional recurrent neural networks
Mike Schuster and Kuldip K. Paliwal · 1997
Earlier work this paper cites.
Recurrent nets that time and count
Felix A. Gers and Jürgen Schmidhuber · 2000
Earlier work this paper cites.
Learning to forget: Continual prediction with LSTM
Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins · 2000
Earlier work this paper cites.
Long short-term memory in recurrent neural networks
Felix A. Gers · 2001
Earlier work this paper cites.
Gradient flow in recurrent nets: the difficulty of learning long-term dependencies
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber · 2001
Earlier work this paper cites.
BLEU: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002
Earlier work this paper cites.

Similar

The principled design of large-scale recursive neural network architectures–DAG-RNNs and the protein structure prediction problem
Pierre Baldi and Gianluca Pollastri · 2003
Cited alongside, same era.
A neural probabilistic language model
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin · 2003
Cited alongside, same era.
METEOR: An automatic metric for MT evaluation with improved correlation with human judgments
Satanjeev Banerjee and Alon Lavie · 2005
Cited alongside, same era.
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Alex Graves and Jürgen Schmidhuber · 2005
Cited alongside, same era.
A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks
Marcus Liwicki, Alex Graves, Horst Bunke, and Jürgen Schmidhuber · 2007
Cited alongside, same era.
Lazy sparse stochastic gradient descent for regularized multinomial logistic regression
Bob Carpenter · 2008
Cited alongside, same era.
Evolving memory cell structures for sequence learning
Justin Bayer, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber · 2009
Cited alongside, same era.
A novel connectionist system for unconstrained handwriting recognition
Alex Graves, Marcus Liwicki, Santiago Fernández, Roman Bertolami, Horst Bunke, and Jürgen Schmidhuber · 2009
Cited alongside, same era.
Sparse online learning via truncated gradient
John Langford, Lihong Li, and Tong Zhang · 2009
Cited alongside, same era.
Efficient learning using forward-backward splitting
Yoram Singer and John C. Duchi · 2009
Cited alongside, same era.
Theano: a CPU and GPU math expression compiler
James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio · 2010
Cited alongside, same era.
Rectified linear units improve restricted Boltzmann machines
Vinod Nair and Geoffrey E. Hinton · 2010
Cited alongside, same era.

Then

Advances in optimizing recurrent networks
Yoshua Bengio, Nicolas Boulanger-Lewandowski, and Razvan Pascanu · 2013
Later among the works it cites.
Learning hierarchical features for scene labeling
Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun · 2013
Later among the works it cites.
Optimizing neural networks with genetic algorithms
Steven A. Harp and Tariq Samad · 2013
Later among the works it cites.
Efficient estimation of word representations in vector space
Original
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean · 2013
Later among the works it cites.
On the importance of initialization and momentum in deep learning
Ilya Sutskever, James Martens, George Dahl, and Geoffrey E. Hinton · 2013
Later among the works it cites.
On rectified linear units for speech processing
Matthew D. Zeiler, M. Ranzato, Rajat Monga, M. Mao, K. Yang, Quoc V. Le, Patrick Nguyen, A. Senior, Vincent Vanhoucke, Jeffrey Dean, et al · 2013
Later among the works it cites.
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio · 2014
Later among the works it cites.
word2vec explained: deriving Mikolov et al.’s negative-sampling word-embedding method
Original
Yoav Goldberg and Omer Levy · 2014
Later among the works it cites.
Neural Turing machines
Original
Alex Graves, Greg Wayne, and Ivo Danihelka · 2014
Later among the works it cites.
Caffe: Convolutional architecture for fast feature embedding
Original
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell · 2014
Later among the works it cites.
Deep visual-semantic alignments for generating image descriptions
Original
Andrej Karpathy and Li Fei-Fei · 2014
Later among the works it cites.
Optimal thresholding of classifiers to maximize F1 measure
Zachary C. Lipton, Charles Elkan, and Balakrishnan Naryanaswamy · 2014
Later among the works it cites.
Deep captioning with multimodal recurrent neural networks (m-RNN)
Original
Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan Yuille · 2014
Later among the works it cites.
Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher D. Manning · 2014
Later among the works it cites.
Grounded compositional semantics for finding and describing images with sentences
Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng · 2014
Later among the works it cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le · 2014
Later among the works it cites.
Learning to execute
Original
Wojciech Zaremba and Ilya Sutskever · 2014
Later among the works it cites.
A survey on the application of recurrent neural networks to statistical language modeling
Wim De Mulder, Steven Bethard, and Marie-Francine Moens · 2015
Closest in time.
Learning meanings for sentences
Charles Elkan · 2015
Closest in time.
The unreasonable effectiveness of recurrent neural networks
Andrej Karpathy · 2015
Closest in time.
Efficient elastic net regularization for sparse linear models
Original
Zachary C. Lipton and Charles Elkan · 2015
Closest in time.
Unsupervised learning of video representations using LSTMs
Original
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov · 2015
Closest in time.
Sequence to sequence–video to text
Original
Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko · 2015
Closest in time.
Show and tell: A neural image caption generator
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan · 2015
Closest in time.
Backpropagation — Wikipedia, the free encyclopedia, 2015
Wikipedia · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…