2016

A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition

Zeyer, Albert, Doetsch, Patrick, Voigtlaender, Paul et al.

Understand

We present a comprehensive study of deep bidirectional long short-term memory (LSTM) recurrent neural network (RNN) based acoustic models for automatic speech recognition (ASR).

We study the effect of size and depth and train models of up to 8 layers.
We investigate the training aspect and study different variants of optimization methods, batching, truncated backpropagation, different regularization techniques such as dropout and $L_2$ regularization, and different gradient clipping variants.
The major part of the experimental analysis was performed on the Quaero corpus.

Built on

“Some methods of speeding up the convergence of iteration methods,”
Boris T Polyak, · 1964
Earlier work this paper cites.
“An application of recurrent nets to phone probability estimation,”
Anthony J Robinson, · 1994
Earlier work this paper cites.
Connectionist speech recognition: a hybrid approach
Hervé Bourlard and Nelson Morgan, · 1994
Earlier work this paper cites.
“Long short-term memory,”
Sepp Hochreiter and Jürgen Schmidhuber, · 1997
Earlier work this paper cites.
“Bidirectional recurrent neural networks,”
Mike Schuster and Kuldip K Paliwal, · 1997
Earlier work this paper cites.
“Learning precise timing with LSTM recurrent networks,”
Felix A Gers, Nicol N Schraudolph, and Jürgen Schmidhuber, · 2003
Earlier work this paper cites.
“Framewise phoneme classification with bidirectional lstm and other neural network architectures,”
Alex Graves and Jürgen Schmidhuber, · 2005
Earlier work this paper cites.
“Gammatone features and feature combination for large vocabulary speech recognition,”
Ralf Schlüter, L Bezrukov, Hannes Wagner, and Hermann Ney, · 2007
Earlier work this paper cites.
“The RWTH 2009 Quaero ASR evaluation system for English and German,”
Markus Nußbaum-Thom, Simon Wiesler, Martin Sundermeyer, Christian Plahl, Stefan Hahn, Ralf Schlüter, and Hermann Ney, · 2010
Earlier work this paper cites.
“Understanding the difficulty of training deep feedforward neural networks,”
Xavier Glorot and Yoshua Bengio, · 2010
Earlier work this paper cites.
“RASR - the RWTH Aachen university open source speech recognition toolkit,”
David Rybach, Stefan Hahn, Patrick Lehnen, David Nolden, Martin Sundermeyer, Zoltan Tüske, Simon Wiesler, Ralf Schlüter, and Hermann Ney, · 2011
Earlier work this paper cites.
“Adaptive subgradient methods for online learning and stochastic optimization,”
John Duchi, Elad Hazan, and Yoram Singer, · 2011
Earlier work this paper cites.
“Feature engineering in context-dependent deep neural networks for conversational speech transcription,”
Frank Seide, Gang Li, Xie Chen, and Dong Yu, · 2011
Earlier work this paper cites.
“Theano: new features and speed improvements,” Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012
Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio, · 2012
Earlier work this paper cites.
“Adadelta: An adaptive learning rate method,”
Original
Matthew D Zeiler, · 2012
Earlier work this paper cites.
“Lecture 6.5 - RMSprop: Divide the gradient by a running average of its recent magnitude.,” COURSERA: Neural Networks for Machine Learning, 2012
Tijmen Tieleman and Geoffrey Hinton, · 2012
Earlier work this paper cites.

Similar

“Improving neural networks by preventing co-adaptation of feature detectors,”
Original
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov, · 2012
Cited alongside, same era.
“Hybrid speech recognition with deep bidirectional LSTM,”
Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed, · 2013
Cited alongside, same era.
“On the importance of initialization and momentum in deep learning,”
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton, · 2013
Cited alongside, same era.
“Deep learning in neural networks: An overview,”
Original
J. Schmidhuber, · 2014
Cited alongside, same era.
“Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition,”
Original
Haşim Sak, Andrew Senior, and Françoise Beaufays, · 2014
Cited alongside, same era.
“Robust speech recognition using long short-term memory recurrent neural networks for hybrid acoustic modelling,”
Jürgen T Geiger, Zixing Zhang, Felix Weninger, Björn Schuller, and Gerhard Rigoll, · 2014
Cited alongside, same era.
“Empirical evaluation of gated recurrent neural networks on sequence modeling,”
Original
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio, · 2014
Cited alongside, same era.
“RASR/NN: The RWTH neural network toolkit for speech recognition,”
Simon Wiesler, Alexander Richard, Pavel Golik, Ralf Schlüter, and Hermann Ney, · 2014
Cited alongside, same era.
“Adam: A method for stochastic optimization,”
Original
Diederik Kingma and Jimmy Ba, · 2014
Cited alongside, same era.
“Mean-normalized stochastic gradient for large-scale deep learning,”
Simon Wiesler, Alexander Richard, Ralf Schlüter, and Hermann Ney, · 2014
Cited alongside, same era.
“Adasecant: robust adaptive secant method for stochastic gradient,”
Original
Caglar Gulcehre, Marcin Moczulski, and Yoshua Bengio, · 2014
Cited alongside, same era.
“Constructing long short-term memory based deep recurrent neural networks for large vocabulary speech recognition,”
Xiangang Li and Xihong Wu, · 2015
Cited alongside, same era.

Then

“Highway long short-term memory RNNs for distant speech recognition,”
Original
Yu Zhang, Guoguo Chen, Dong Yu, Kaisheng Yao, Sanjeev Khudanpur, and James Glass, · 2015
Later among the works it cites.
“An empirical exploration of recurrent network architectures,”
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever, · 2015
Later among the works it cites.
“LSTM: A search space odyssey,”
Original
Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber, · 2015
Later among the works it cites.
“Benchmarking of LSTM networks,”
Original
Thomas M Breuel, · 2015
Later among the works it cites.
“Training very deep networks,”
Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber, · 2015
Later among the works it cites.
“Deep residual learning for image recognition,”
Original
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, · 2015
Later among the works it cites.
“Incorporating Nesterov momentum into Adam,”
Timothy Dozat, · 2015
Later among the works it cites.
“Adding gradient noise improves learning for very deep networks,”
Original
A. Neelakantan, L. Vilnis, Q. V. Le, I. Sutskever, L. Kaiser, K. Kurach, and J. Martens, · 2015
Later among the works it cites.
“RMSprop loses to SMORMS3 - beware the epsilon!,”
Simon Funk, · 2015
Later among the works it cites.
“Speaker adaptive joint training of gaussian mixture models and bottleneck features,”
Zoltán Tüske, Pavel Golik, Ralf Schlüter, and Hermann Ney, · 2015
Later among the works it cites.
“Multilingual features based keyword search for very low-resource languages,”
Original
Pavel Golik, Zoltán Tüske, Ralf Schlüter, and Hermann Ney, · 2015
Later among the works it cites.
“GitHub repository with config files for LSTM experiments in RETURNN,”
2016
Closest in time.
“RETURNN: The RWTH extensible training framework for universal recurrent neural networks,”
Original
Patrick Doetsch, Albert Zeyer, Paul Voigtlaender, Ilya Kulikov, Ralf Schlüter, and Hermann Ney, · 2016
Closest in time.
“Towards online-recognition with deep bidirectional LSTM acoustic models,”
Albert Zeyer, Ralf Schlüter, and Hermann Ney, · 2016
Closest in time.
“Associative long short-term memory,”
Original
Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, and Alex Graves, · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…