2017

Recent Advances in Recurrent Neural Networks

Salehinejad, Hojjat, Sankar, Sharan, Barfett, Joseph et al.

Understand

Recurrent neural networks (RNNs) are capable of learning features and long term dependencies from sequential and time-series data.

The RNNs have a stack of non-linear units where at least one connection between units forms a directed cycle.
A well-trained RNN can model any dynamical system; however, training RNNs is mostly plagued by issues in learning long-term dependencies.
In this paper, we present a survey on RNNs and several new advances for newcomers and professionals in the field.

Built on

P. Potash, A. Romanov, and A. Rumshisky, “Ghostwriter: using an lstm for automatic rap lyric generation,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , 2015, pp. 1919– 1924
1924
Earlier work this paper cites.
L. Theis and M. Bethge, “Generative image modeling using spatial lstms,” in Advances in Neural Information Processing Systems , 2015, pp. 1918–1926
1926
Earlier work this paper cites.
B. T. Polyak, “Some methods of speeding up the convergence of iteration methods,” USSR Computational Mathematics and Mathematical Physics , vol. 4, no. 5, pp. 1–17, 1964
1964
Earlier work this paper cites.
A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang, “Phoneme recognition using time-delay neural networks,” IEEE transactions on acoustics, speech, and signal processing , vol. 37, no. 3, pp. 328–339, 1989
1989
Earlier work this paper cites.
P. J. Werbos, “Backpropagation through time: what it does and how to do it,” Proceedings of the IEEE , vol. 78, no. 10, pp. 1550–1560, 1990
1990
Earlier work this paper cites.
R. J. Williams, “Training recurrent networks using the extended kalman filter,” in Neural Networks, 1992. IJCNN., International Joint Conference on , vol. 4. IEEE, 1992, pp. 241–246
1992
Earlier work this paper cites.
C. Tuerk and T. Robinson, “Speech synthesis using artificial neural networks trained on cepstral coefficients.” in EUROSPEECH , 1993
1993
Earlier work this paper cites.
Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE transactions on neural networks , vol. 5, no. 2, pp. 157–166, 1994
1994
Earlier work this paper cites.
S. Haykin, Neural networks: a comprehensive foundation . Prentice Hall PTR, 1994
1994
Earlier work this paper cites.
G. V. Puskorius and L. A. Feldkamp, “Neurocontrol of nonlinear dynamical systems with kalman filter trained recurrent networks,” IEEE Transactions on neural networks , vol. 5, no. 2, pp. 279–297, 1994
1994
Earlier work this paper cites.
P. J. Angeline, G. M. Saunders, and J. B. Pollack, “An evolutionary algorithm that constructs recurrent neural networks,” IEEE transactions on Neural Networks , vol. 5, no. 1, pp. 54–65, 1994
1994
Earlier work this paper cites.
K. Unnikrishnan and K. P. Venugopal, “Alopex: A correlation-based learning algorithm for feedforward and recurrent neural networks,” Neural Computation , vol. 6, no. 3, pp. 469–490, 1994
1994
Earlier work this paper cites.
R. J. Williams and D. Zipser, “Gradient-based learning algorithms for recurrent networks and their computational complexity,” Backpropagation: Theory, architectures, and applications , vol. 1, pp. 433–486, 1995
1995
Earlier work this paper cites.
M. Schuster, “Bi-directional recurrent neural networks for speech recognition,” Technical report, Tech. Rep., 1996
1996
Earlier work this paper cites.
M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Transactions on Signal Processing , vol. 45, no. 11, pp. 2673–2681, 1997
1997
Earlier work this paper cites.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.
S. Ma and C. Ji, “A unified approach on fast training of feedforward and recurrent networks using em algorithm,” IEEE transactions on signal processing , vol. 46, no. 8, pp. 2270–2274, 1998
1998
Earlier work this paper cites.
O. Karaali, G. Corrigan, I. Gerson, and N. Massey, “Text-to-speech conversion with neural networks: A recurrent tdnn approach,” arXiv preprint cs/9811032 , 1998
1998
Earlier work this paper cites.
L.-W. Chan and C.-C. Szeto, “Training recurrent network with block-diagonal approximated levenberg-marquardt algorithm,” in Neural Networks, 1999. IJCNN’99. International Joint Conference on , vol. 3. IEEE, 1999, pp. 1521–1526
1999
Earlier work this paper cites.
F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with lstm,” 1999
1999
Earlier work this paper cites.
S. S. Haykin et al. , Kalman filtering and neural networks . Wiley Online Library, 2001
2001
Earlier work this paper cites.
D. Eck and J. Schmidhuber, “A first look at music composition using lstm recurrent neural networks,” Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale , vol. 103, 2002
2002
Earlier work this paper cites.
——, “Finding temporal structure in music: Blues improvisation with lstm recurrent networks,” in Neural Networks for Signal Processing, 2002. Proceedings of the 2002 12th IEEE Workshop on . IEEE, 2002, pp. 747–756
2002
Earlier work this paper cites.
J. A. Pérez-Ortiz, F. A. Gers, D. Eck, and J. Schmidhuber, “Kalman filters improve lstm network performance in problems unsolvable by traditional recurrent nets,” Neural Networks , vol. 16, no. 2, pp. 241–250, 2003
2003
Earlier work this paper cites.
P. Baldi and G. Pollastri, “The principled design of large-scale recursive neural network architectures–dag-rnns and the protein structure prediction problem,” Journal of Machine Learning Research , vol. 4, no. Sep, pp. 575–602, 2003
2003
Earlier work this paper cites.
L. Bottou, “Stochastic learning,” in Advanced lectures on machine learning . Springer, 2004, pp. 146–168
2004
Earlier work this paper cites.
A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional lstm and other neural network architectures,” Neural Networks , vol. 18, no. 5, pp. 602–610, 2005
2005
Earlier work this paper cites.
G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural computation , vol. 18, no. 7, pp. 1527–1554, 2006
2006
Earlier work this paper cites.
C. M. Bishop, Pattern recognition and machine learning . springer, 2006
2006
Earlier work this paper cites.
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning . ACM, 2006, pp. 369–376
2006
Earlier work this paper cites.
Y. Bengio, Y. LeCun et al. , “Scaling learning algorithms towards ai,” Large-scale kernel machines , vol. 34, no. 5, pp. 1–41, 2007
2007
Earlier work this paper cites.
Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in Advances in neural information processing systems , 2007, pp. 153–160
2007
Earlier work this paper cites.
A. Graves, S. Fernandez, and J. Schmidhuber, “Multi-dimensional recurrent neural networks,” arXiv preprint arXiv:0705.2011v1 , 2007
Original
2007
Earlier work this paper cites.
M. Wöllmer, F. Eyben, S. Reiter, B. Schuller, C. Cox, E. Douglas-Cowie, and R. Cowie, “Abandoning emotion classes-towards continuous emotion recognition with modelling of long-range dependencies,” in Ninth Annual Conference of the International Speech Communication Association , 2008
2008
Earlier work this paper cites.
A. Graves, M. Liwicki, H. Bunke, J. Schmidhuber, and S. Fernández, “Unconstrained on-line handwriting recognition with recurrent neural networks,” in Advances in Neural Information Processing Systems , 2008, pp. 577–584
2008
Earlier work this paper cites.
A. Graves and J. Schmidhuber, “Offline handwriting recognition with multidimensional recurrent neural networks,” in Advances in neural information processing systems , 2009, pp. 545–552
2009
Earlier work this paper cites.
T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur, “Recurrent neural network based language model.” in Interspeech , vol. 2, 2010, p. 3
2010
Earlier work this paper cites.
J. Martens, “Deep learning via hessian-free optimization,” in Proceedings of the 27th International Conference on Machine Learning (ICML-10) , 2010, pp. 735–742
2010
Earlier work this paper cites.
D. T. Mirikitani and N. Nikolaev, “Recursive bayesian recurrent neural networks for time-series modeling,” IEEE Transactions on Neural Networks , vol. 21, no. 2, pp. 262–274, 2010
2010
Earlier work this paper cites.
I. Sutskever, J. Martens, and G. E. Hinton, “Generating text with recurrent neural networks,” in Proceedings of the 28th International Conference on Machine Learning (ICML-11) , 2011, pp. 1017–1024
2011
Earlier work this paper cites.
A. Cotter, O. Shamir, N. Srebro, and K. Sridharan, “Better mini-batch algorithms via accelerated gradient methods,” in Advances in neural information processing systems , 2011, pp. 1647–1655
2011
Earlier work this paper cites.
J. Martens and I. Sutskever, “Learning recurrent neural networks with hessian-free optimization,” in Proceedings of the 28th International Conference on Machine Learning (ICML-11) , 2011, pp. 1033–1040
2011
Earlier work this paper cites.
A. Graves, S. Fernández, and J. Schmidhuber, “Multi-dimensional recurrent neural networks,” 2007. [Online]. Available: http://arxiv.org/abs/0705.2011
Original
2011
Earlier work this paper cites.

Similar

T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Černockỳ, “Empirical evaluation and combination of advanced language modeling techniques,” in Twelfth Annual Conference of the International Speech Communication Association , 2011
2011
Cited alongside, same era.
T. Mikolov, S. Kombrink, L. Burget, J. Černockỳ, and S. Khudanpur, “Extensions of recurrent neural network language model,” in Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on . IEEE, 2011, pp. 5528–5531
2011
Cited alongside, same era.
M. El Ayadi, M. S. Kamel, and F. Karray, “Survey on speech emotion recognition: Features, classification schemes, and databases,” Pattern Recognition , vol. 44, no. 3, pp. 572–587, 2011
2011
Cited alongside, same era.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems , 2012, pp. 1097–1105
2012
Cited alongside, same era.
T. Mikolov, I. Sutskever, A. Deoras, H.-S. Le, S. Kombrink, and J. Cernocky, “Subword language modeling with neural networks,” preprint (http://www. fit. vutbr. cz/imikolov/rnnlm/char. pdf) , 2012
2012
Cited alongside, same era.
T. Mikolov and G. Zweig, “Context dependent recurrent neural network language model.” SLT , vol. 12, pp. 234–239, 2012
2012
Cited alongside, same era.
Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, “Efficient backprop,” in Neural networks: Tricks of the trade . Springer, 2012, pp. 9–48
2012
Cited alongside, same era.
J. Martens and I. Sutskever, “Training deep and recurrent networks with hessian-free optimization,” Neural networks: Tricks of the trade , pp. 479–535, 2012
2012
Cited alongside, same era.
O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, and G. Penn, “Applying convolutional neural networks concepts to hybrid nn-hmm model for speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on . IEEE, 2012, pp. 4277–4280
2012
Cited alongside, same era.
A. Graves, Supervised sequence labelling with recurrent neural networks . Springer Science & Business Media, 2012, vol. 385
2012
Cited alongside, same era.
M. Sundermeyer, R. Schl"uter, and H. Ney, “Lstm neural networks for language modeling,” in Thirteenth Annual Conference of the International Speech Communication Association , 2012
2012
Cited alongside, same era.
H. A. Bourlard and N. Morgan, Connectionist speech recognition: a hybrid approach . Springer Science & Business Media, 2012, vol. 247
2012
Cited alongside, same era.

Then

N. Kalchbrenner, I. Danihelka, and A. Graves, “Grid long short-term memory,” arXiv preprint arXiv:1507.01526 , 2015
Original
2015
Later among the works it cites.
K. Yao, T. Cohn, K. Vylomova, K. Duh, and C. Dyer, “Depth-gated lstm,” arXiv preprint arXiv:1508.03790 , 2015
Original
2015
Later among the works it cites.
M. Ballesteros, C. Dyer, and N. A. Smith, “Improved transition-based parsing by modeling characters instead of words with lstms,” arXiv preprint arXiv:1508.00657 , 2015
Original
2015
Later among the works it cites.
S. Wang and J. Jiang, “Learning natural language inference with lstm,” arXiv preprint arXiv:1512.08849 , 2015
Original
2015
Later among the works it cites.
V. Veeriah, N. Zhuang, and G.-J. Qi, “Differential recurrent neural networks for action recognition,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 4041–4049
2015
Later among the works it cites.
X. Liang, X. Shen, D. Xiang, J. Feng, L. Lin, and S. Yan, “Semantic object parsing with local-global long short-term memory,” arXiv preprint arXiv:1511.04510 , 2015
Original
2015
Later among the works it cites.
J. Li, A. Mohamed, G. Zweig, and Y. Gong, “Lstm time and frequency recurrence for automatic speech recognition,” 2015
2015
Later among the works it cites.
X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” arXiv preprint arXiv:1506.04214 , 2015
Original
2015
Later among the works it cites.
J. Weston, A. Bordes, S. Chopra, and T. Mikolov, “Towards ai-complete question answering: a set of prerequisite toy tasks,” arXiv preprint arXiv:1502.05698 , 2015
Original
2015
Later among the works it cites.
A. Kumar, O. Irsoy, J. Su, J. Bradbury, R. English, B. Pierce, P. Ondruska, I. Gulrajani, and R. Socher, “Ask me anything: Dynamic memory networks for natural language processing,” arXiv preprint arXiv:1506.07285 , 2015
Original
2015
Later among the works it cites.
S. Sukhbaatar, J. Weston, R. Fergus et al. , “End-to-end memory networks,” in Advances in Neural Information Processing Systems , 2015, pp. 2431–2439
2015
Later among the works it cites.
T. Moon, H. Choi, H. Lee, and I. Song, “Rnndrop: A novel dropout for rnns in asr,” in Automatic Speech Recognition and Understanding (ASRU), 2015 IEEE Workshop on . IEEE, 2015, pp. 65–70
2015
Later among the works it cites.
D. Krueger and R. Memisevic, “Regularizing rnns by stabilizing activations,” arXiv preprint arXiv:1511.08400 , 2015
Original
2015
Later among the works it cites.
D. Tang, B. Qin, and T. Liu, “Document modeling with gated recurrent neural network for sentiment classification.” in EMNLP , 2015, pp. 1422– 1432
2015
Later among the works it cites.
S. Lai, L. Xu, K. Liu, and J. Zhao, “Recurrent convolutional neural networks for text classification.” in AAAI , vol. 333, 2015, pp. 2267– 2273
2015
Later among the works it cites.
N. Jaitly, D. Sussillo, Q. V. Le, O. Vinyals, I. Sutskever, and S. Bengio, “A neural transducer,” arXiv preprint arXiv:1511.04868 , 2015
Original
2015
Later among the works it cites.
J. Lee and I. Tashev, “High-level feature representation using recurrent neural network for speech emotion recognition.” in INTERSPEECH , 2015, pp. 1537–1540
2015
Later among the works it cites.
K. Fan, Z. Wang, J. Beck, J. Kwok, and K. A. Heller, “Fast second order stochastic backpropagation for variational inference,” in Advances in Neural Information Processing Systems , 2015, pp. 1387–1395
2015
Later among the works it cites.
H. Zen and H. Sak, “Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis,” in Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on . IEEE, 2015, pp. 4470–4474
2015
Later among the works it cites.
W. Byeon, T. M. Breuel, F. Raue, and M. Liwicki, “Scene labeling with lstm recurrent neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 3547–3555
2015
Later among the works it cites.
K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra, “Draw: A recurrent neural network for image generation,” arXiv preprint arXiv:1502.04623 , 2015
Original
2015
Later among the works it cites.
J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 2625–2634
2015
Later among the works it cites.
L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville, “Video description generation incorporating spatio-temporal features and a soft-attention mechanism,” arXiv preprint arXiv:1502.08029 , 2015
Original
2015
Later among the works it cites.
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning . MIT Press, 2016, http://www.deeplearningbook.org
2016
Later among the works it cites.
S. Ruder, “An overview of gradient descent optimization algorithms,” arXiv preprint arXiv:1609.04747 , 2016
Original
2016
Later among the works it cites.
T. Tanaka, T. Moriya, T. Shinozaki, S. Watanabe, T. Hori, and K. Duh, “Evolutionary optimization of long short-term memory neural network language model,” The Journal of the Acoustical Society of America , vol. 140, no. 4, pp. 3062–3062, 2016
2016
Later among the works it cites.
K. Tran, A. Bisazza, and C. Monz, “Recurrent memory network for language modeling,” arXiv preprint arXiv:1601.01272 , 2016
Original
2016
Later among the works it cites.
M. Arjovsky, A. Shah, and Y. Bengio, “Unitary evolution recurrent neural networks,” in International Conference on Machine Learning , 2016, pp. 1120–1128
2016
Later among the works it cites.
S. Semeniuta, A. Severyn, and E. Barth, “Recurrent dropout without memory loss,” arXiv preprint arXiv:1603.05118 , 2016
Original
2016
Later among the works it cites.
D. Krueger, T. Maharaj, J. Kramár, M. Pezeshki, N. Ballas, N. R. Ke, A. Goyal, Y. Bengio, H. Larochelle, A. Courville et al. , “Zoneout: Regularizing rnns by randomly preserving hidden activations,” arXiv preprint arXiv:1606.01305 , 2016
Original
2016
Later among the works it cites.
M. Ghazvininejad, X. Shi, Y. Choi, and K. Knight, “Generating topical poetry.” in EMNLP , 2016, pp. 1183– 1191
2016
Later among the works it cites.
D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on . IEEE, 2016, pp. 4945–4949
2016
Later among the works it cites.
G. Trigeorgis, F. Ringeval, R. Brueckner, E. Marchi, M. A. Nicolaou, B. Schuller, and S. Zafeiriou, “Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on . IEEE, 2016, pp. 5200–5204
2016
Later among the works it cites.
A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499 , 2016
Original
2016
Later among the works it cites.
B. Shuai, Z. Zuo, B. Wang, and G. Wang, “Dag-recurrent neural networks for scene labeling,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 3620–3629
2016
Later among the works it cites.
A. v. d. Oord, N. Kalchbrenner, and K. Kavukcuoglu, “Pixel recurrent neural networks,” arXiv preprint arXiv:1601.06759 , 2016
Original
2016
Later among the works it cites.
H. Salehinejad, S. Rahnamayan, and H. R. Tizhoosh, “Micro-differential evolution: Diversity enhancement and a comparative study,” Applied Soft Computing , vol. 52, pp. 812–833, 2017
2017
Closest in time.
L. Jing, C. Gulcehre, J. Peurifoy, Y. Shen, M. Tegmark, M. Soljačić, and Y. Bengio, “Gated orthogonal recurrent units: On learning to forget,” arXiv preprint arXiv:1706.02761 , 2017
Original
2017
Closest in time.
H. Salehinejad, J. Barfett, S. Valaee, E. Colak, A. Mnatzakanian, and T. Dowdell, “Interpretation of mammogram and chest radiograph reports using deep neural networks-preliminary results,” arXiv preprint arXiv:1708.09254 , 2017
Original
2017
Closest in time.
M. Mardani, H. Monajemi, V. Papyan, S. Vasanawala, D. Donoho, and J. Pauly, “Recurrent generative adversarial networks for proximal learning and automated compressive image recovery,” arXiv preprint arXiv:1711.10046 , 2017
Original
2017
Closest in time.
H. Salehinejad, S. Valaee, T. Dowdell, E. Colak, and J. Barfett, “Generalization of deep neural networks for chest pathology classification in x-rays using generative adversarial networks,” arXiv preprint arXiv:1712.01636 , 2017
Original
2017
Closest in time.
H. Salehinejad, S. Rahnamayan, H. R. Tizhoosh, and S. Y. Chen, “Micro-differential evolution with vectorized random mutation factor,” in Evolutionary Computation (CEC), 2014 IEEE Congress on . IEEE, 2014, pp. 2055–2062
2062
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…