2019

Transformer-based Acoustic Modeling for Hybrid Speech Recognition

Wang, Yongqiang, Mohamed, Abdelrahman, Le, Duc et al.

Understand

We propose and evaluate transformer-based acoustic models (AMs) for hybrid speech recognition.

Several modeling choices are discussed in this work, including various positional embedding methods and an iterated loss to enable training deep transformers.
We also present a preliminary study of using limited right context in transformer models, which makes it possible for streaming applications.
We demonstrate that on the widely used Librispeech benchmark, our transformer-based AM outperforms the best published hybrid result by 19% to 26% relative when the standard n-gram language model (LM) is used.

Built on

“Subphonetic modeling with markov states-senone,”
M.-Y. Hwang and X. Huang, · 1992
Earlier work this paper cites.
“Learning long-term dependencies with gradient descent is difficult,”
Y. Bengio, P. Simard, et al., · 1994
Earlier work this paper cites.
“Long short-term memory,”
S. Hochreiter and J. Schmidhuber, · 1997
Earlier work this paper cites.
“Conversational speech transcription using context-dependent deep neural networks,”
F. Seide, G. Li, and D. Yu, · 2011
Earlier work this paper cites.
“The kaldi speech recognition toolkit,”
D. Povey, A. Ghoshal, G. Boulianne, et al., · 2011
Earlier work this paper cites.
“Deep neural networks for acoustic modeling in speech recognition,”
G. Hinton, L. Deng, D. Yu, et al., · 2012
Earlier work this paper cites.
“Sequence-discriminative training of deep neural networks.,”
K. Veselỳ, A. Ghoshal, L. Burget, and D. Povey, · 2013
Earlier work this paper cites.
“Long short-term memory recurrent neural network architectures for large scale acoustic modeling,”
H. Sak, A. Senior, and F. Beaufays, · 2014
Earlier work this paper cites.
“Convolutional neural networks for speech recognition,”
O. Abdel-Hamid, A. Mohamed, H. Jiang, et al., · 2014
Earlier work this paper cites.
“Adam: A method for stochastic optimization,”
Original
D. P. Kingma and J. Ba, · 2014
Earlier work this paper cites.
“Very deep convolutional networks for large-scale image recognition,”
Original
K. Simonyan and A. Zisserman, · 2014
Earlier work this paper cites.
“A time delay neural network architecture for efficient modeling of long temporal contexts,”
V. Peddinti, D. Povey, and S. Khudanpur, · 2015
Earlier work this paper cites.
“Feedforward sequential memory neural networks without recurrent feedback,”
Original
S. Zhang, H. Jiang, S. Wei, and L. Dai, · 2015
Earlier work this paper cites.
“Librispeech: an asr corpus based on public domain audio books,”
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, · 2015
Earlier work this paper cites.

Similar

“Audio augmentation for speech recognition,”
T. Ko, V. Peddinti, D. Povey, et al., · 2015
Cited alongside, same era.
“End-to-end attention-based large vocabulary speech recognition,”
D. Bahdanau, J. Chorowski, D. Serdyuk, et al., · 2016
Cited alongside, same era.
“Wav2letter: an end-to-end convnet-based speech recognition system,”
Original
R. Collobert, C. Puhrsch, and G. Synnaeve, · 2016
Cited alongside, same era.
“Layer normalization,”
Original
J. Lei Ba, J. Kiros, and G. E. Hinton, · 2016
Cited alongside, same era.
“Gaussian error linear units (gelus),”
Original
D. Hendrycks and K. Gimpel, · 2016
Cited alongside, same era.
“Highway long short-term memory RNNs for distant speech recognition,”
Y. Zhang, G. Chen, D. Yu, et al., · 2016
Cited alongside, same era.
“Attention is all you need,”
A. Vaswani, N. Shazeer, N. Parmar, et al., · 2017
Cited alongside, same era.
“Very deep convolutional networks for end-to-end speech recognition,”
Y. Zhang, W. Chan, and N. Jaitly, · 2017
Cited alongside, same era.
“State-of-the-art speech recognition with sequence-to-sequence models,”
C.-C. Chiu, T.N. Sainath, Y. Wu, et al., · 2018
Cited alongside, same era.
“Bert: Pre-training of deep bidirectional transformers for language understanding,”
Original
J. Devlin, M.-W. Chang, K. Lee, et al., · 2018
Cited alongside, same era.
“Improving language understanding by generative pre-training,”
A. Radford, K. Narasimhan, Tim S., et al., · 2018
Cited alongside, same era.
“Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition,”
L. Dong, S. Xu, and B. Xu, · 2018
Cited alongside, same era.

Then

“A time-restricted self-attention layer for asr,”
D. Povey, Hossein Hadian, P. Ghahremani, et al., · 2018
Later among the works it cites.
“The best of both worlds: Combining recent advances in neural machine translation,”
Original
M. Chen, O. Firat, A. Bapna, et al., · 2018
Later among the works it cites.
“Specaugment: A simple data augmentation method for automatic speech recognition,”
Original
D. S. Park, W. Chan, Y. Zhang, et al., · 2019
Closest in time.
“Streaming end-to-end speech recognition for mobile devices,”
Y. He, T. N. Sainath, R. Prabhavalkar, et al., · 2019
Closest in time.
“Self-attention networks for connectionist temporal classification in speech recognition,”
J. Salazar, K. Kirchhoff, and Z. Huang, · 2019
Closest in time.
“From senones to chenones: Tied context-dependent graphemes for hybrid speech recognition,”
Original
D. Le, X. Zhang, W. Zheng, et al., · 2019
Closest in time.
“Transformers with convolutional context for asr,”
Original
A. Mohamed, D. Okhonko, and L. Zettlemoyer, · 2019
Closest in time.
“A Comparative Study on Transformer vs RNN in Speech Applications,”
Original
S. Karita, N. Chen, T. Hayashi, et al., · 2019
Closest in time.
“Character-level language modeling with deeper self-attention,”
R. Al-Rfou, D. Choe, N. Constant, et al., · 2019
Closest in time.
“fairseq: A Fast, Extensible Toolkit for Sequence Modeling,”
O. Myle, E. Sergey, B. Alexei, F. Angela, et al., · 2019
Closest in time.
“RWTH ASR Systems for LibriSpeech: Hybrid vs Attention-w/o Data Augmentation,”
Original
C. Lüscher, E. Beck, K. Irie, et al., · 2019
Closest in time.
“State-of-the-art speech recognition using multi-stream self-attention with dilated 1d convolutions,”
Original
K. J. Han, R. Prieto, K. Wu, and T. Ma, · 2019
Closest in time.
“Deja-vu: Double feature presentation and iterated loss in deep transformer networks,”
A. Tjandra, C. Liu, F. Zhang, et al., · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…