Fetching the paper…

Reading the bibliography…

2017

Improved training for online end-to-end speech recognition systems

Kim, Suyoun, Seltzer, Michael L., Li, Jinyu et al.

Original
PDF
Post on X

Understand

Achieving high accuracy with end-to-end speech recognizers requires careful parameter initialization prior to training.

Otherwise, the networks may fail to find a good local optimum.
This is particularly true for online networks, such as unidirectional LSTMs.
Currently, the best strategy to train such systems is to bootstrap the training from a tied-triphone system.

Improved training for online end-to-end speech recognition systems · Around

Built on

S. Hochreiter and J. Schmidhuber, “Long short-term memory,”
1997
Earlier work this paper cites.
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in
2006
Earlier work this paper cites.
Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in
2009
Earlier work this paper cites.
T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Černockỳ, “Strategies for training large scale neural network language models,” in
2011
Earlier work this paper cites.
A. Graves, N. Jaitly, and A.-r. Mohamed, “Hybrid speech recognition with deep bidirectional lstm,” in
Original
2013
Earlier work this paper cites.
A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in
2014
Earlier work this paper cites.
A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates
Original
2014
Earlier work this paper cites.
J. Li, R. Zhao, J.-T. Huang, and Y. Gong, “Learning small-size dnn with output-distribution-based criteria,” in
2014
Earlier work this paper cites.
J. Ba and R. Caruana, “Do deep nets really need to be deep?” in
2014
Earlier work this paper cites.
W. Zaremba and I. Sutskever, “Learning to execute,”
Original
2014
Earlier work this paper cites.
Y. Miao, M. Gowayyed, and F. Metze, “EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding,” in
2015
Earlier work this paper cites.
J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in
2015
Earlier work this paper cites.

Similar

W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, attend and spell,”
Original
2015
Cited alongside, same era.
H. Sak, A. Senior, K. Rao, O. Irsoy, A. Graves, F. Beaufays, and J. Schalkwyk, “Learning acoustic frame labeling for speech recognition with recurrent neural networks,” in
2015
Cited alongside, same era.
G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”
Original
2015
Cited alongside, same era.
W. Chan, N. R. Ke, and I. Lane, “Transferring knowledge from a rnn to a dnn,”
Original
2015
Cited alongside, same era.
H. Sak, A. Senior, K. Rao, and F. Beaufays, “Fast and accurate recurrent neural network acoustic models for speech recognition,”
Original

Then

E. Battenberg, R. Child, A. Coates, C. Fougner, Y. Gaur, J. Huang, H. Jun, A. Kannan, M. Kliegl, A. Kumar
Original
2017
Closest in time.
K. Rao and H. Sak, “Multi-accent speech recognition with hierarchical grapheme based models,” in
2017
Closest in time.
K. Audhkhasi, B. Ramabhadran, G. Saon, M. Picheny, and D. Nahamoo, “Direct acoustics-to-word models for english conversational speech recognition,”
Original
2017
Closest in time.
S. Watanabe, T. Hori, J. Le Roux, and J. R. Hershey, “Student-teacher network learning with enhanced features,” in
2017
Closest in time.
J. Li, M. L. Seltzer, X. Wang, R. Zhao, and Y. Gong, “Large-scale domain adaptation via teacher-student learning,”
2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2015

Cited alongside, same era.

D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen

2016

Cited alongside, same era.

A. Senior, H. Sak, and K. Rao, “Flatstart-CTC: a new acoustic model training procedure for speech recognition,” in

2016

Cited alongside, same era.

H. Soltau, H. Liao, and H. Sak, “Neural speech recognizer: Acoustic-to-word lstm model for large vocabulary speech recognition,”

2016

Cited alongside, same era.

Y. Kim and A. M. Rush, “Sequence-level knowledge distillation,”

2016

Cited alongside, same era.

G. Zweig, C. Yu, J. Droppo, and A. Stolcke, “Advances in all-neural speech recognition,” in

2017

Cited alongside, same era.

S. Kim, T. Hori, and S. Watanabe, “Joint ctc-attention based end-to-end speech recognition using multi-task learning,” in

2017

Cited alongside, same era.

T. Hori, S. Watanabe, and J. Hershey, “Joint CTC/attention decoding for end-to-end speech recognition,” in

2017

Cited alongside, same era.

G. Pereyra, G. Tucker, J. Chorowski, L. Kaiser, and G. Hinton, “Regularizing neural networks by penalizing confident output distributions,”

2017

Closest in time.

K. Audhkhasi, B. Kingsbury, B. Ramabhadran, G. Saon, and M. Picheny, “Building competitive direct acoustics-to-word models for english conversational speech recognition,” in

2018

Closest in time.

J. Li, G. Ye, A. Das, R. Zhao, and Y. Gong, “Advancing Acoustic-to-Word CTC Model,” in

2018

Closest in time.

C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, K. Gonina

2018

Closest in time.

A. Das, J. Li, R. Zhao, and Y. Gong, “Advancing connectionist temporal classification with attention modeling,” in

2018

Closest in time.

R. Takashima, S. Li, and H. Kawai, “An investigation of a knowledge distillation method for CTC acoustic models,” in

2018

Closest in time.