2019

From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition

Le, Duc, Zhang, Xiaohui, Zheng, Weiyi et al.

Understand

There is an implicit assumption that traditional hybrid approaches for automatic speech recognition (ASR) cannot directly model graphemes and need to rely on phonetic lexicons to get competitive performance, especially on English which has poor grapheme-phoneme correspondence.

In this work, we show for the first time that, on English, hybrid ASR systems can in fact model graphemes effectively by leveraging tied context-dependent graphemes, i.e., chenones.
Our chenone-based systems significantly outperform equivalent senone baselines by 4.5% to 11.1% relative on three different English datasets.
Our results on Librispeech are state-of-the-art compared to other hybrid approaches and competitive with previously published end-to-end numbers.

Built on

“Tree-based state tying for high accuracy acoustic modelling,”
S. J. Young, J. J. Odell, and P. C. Woodland, · 1994
Earlier work this paper cites.
“Context-dependent acoustic modeling using graphemes for large vocabulary speech recognition,”
S. Kanthak and H. Ney, · 2002
Earlier work this paper cites.
“Grapheme based speech recognition,”
M. Killer, S. Stuker, and T. Schultz, · 2003
Earlier work this paper cites.
“Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,”
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, · 2006
Earlier work this paper cites.
“Joint-sequence models for grapheme-to-phoneme conversion,”
M. Bisani and H. Ney, · 2008
Earlier work this paper cites.
“Revisiting graphemes with increasing amounts of data,”
Y. Sung, T. Hughes, F. Beaufays, and B. Strope, · 2009
Earlier work this paper cites.
“Large Vocabulary Continuous Speech Recognition With Context-Dependent DBN-HMMS,”
G. Dahl, D. Yu, L. Deng, and A. Acero, · 2011
Earlier work this paper cites.
“Acoustic modeling using deep belief networks,”
A. Mohamed, G. E. Dahl, and G. E. Hinton, · 2012
Earlier work this paper cites.
“Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition,”
G. Dahl, D. Yu, L. Deng, and A. Acero, · 2012
Earlier work this paper cites.
“Deep Neural Networks for Acoustic Modeling in Speech Recognition,”
G. Hinton, L. Deng, D. Yu, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, G. Dahl, and B. Kingsbury, · 2012
Earlier work this paper cites.
“Sequence transduction with recurrent neural networks,”
A. Graves, · 2012
Earlier work this paper cites.
“Speech recognition with deep recurrent neural networks,”
A. Graves, A. Mohamed, and G. E. Hinton, · 2013
Earlier work this paper cites.
“Hybrid speech recognition with deep bidirectional lstm,”
A. Graves, N. Jaitly, and A. Mohamed, · 2013
Earlier work this paper cites.
“Long short-term memory recurrent neural network architectures for large scale acoustic modeling,”
H. Sak, A. W. Senior, and F. Beaufays, · 2014
Earlier work this paper cites.
“Sequence Discriminative Distributed Training of Long Short-Term Memory Recurrent Neural Networks,”
H. Sak, O. Vinyals, G. Heigold, A. W. Senior, E. McDermott, R. Monga, and M. Mao, · 2014
Earlier work this paper cites.
“Speech recognition without a lexicon-bridging the gap between graphemic and phonetic systems.,”
D. Harwath and J. Glass, · 2014
Earlier work this paper cites.
“Towards end-to-end speech recognition with recurrent neural networks,”
A. Graves and N. Jaitly, · 2014
Earlier work this paper cites.

Similar

“Adam: A method for stochastic optimization,”
D. P. Kingma and J. Ba, · 2014
Cited alongside, same era.
“Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks,”
T. N. Sainath, O. Vinyals, A. Senior, and H. Sak, · 2015
Cited alongside, same era.
“Unicode-based graphemic systems for limited resource languages,”
M. Gales, K. Knill, and A. Ragni, · 2015
Cited alongside, same era.
“Fast and accurate recurrent neural network acoustic models for speech recognition,”
H. Sak, A. Senior, K. Rao, and F. Beaufays, · 2015
Cited alongside, same era.
“Attention-based models for speech recognition,”
J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, · 2015
Cited alongside, same era.
“Librispeech: an asr corpus based on public domain audio books,”
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, · 2015
Cited alongside, same era.
“Audio augmentation for speech recognition,”
T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, · 2015
Cited alongside, same era.
“Deep speech 2: End-to-end speech recognition in english and mandarin,”
D. Amodei et al., · 2016
Cited alongside, same era.
“Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,”
W. Chan, N. Jaitly, Q. Le, and O. Vinyals, · 2016
Cited alongside, same era.
“Highway long short-term memory rnns for distant speech recognition,”
Y. Zhang, G. Chen, D. Yu, K. Yaco, S. Khudanpur, and J. Glass, · 2016
Cited alongside, same era.
“Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering,”
K. Chen and Q. Huo, · 2016
Cited alongside, same era.
“Purely sequence-trained neural networks for ASR based on lattice-free MMI,”
D. Povey, V. Peddinti, D. Galvez, P. Ghahrmani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, · 2016
Cited alongside, same era.

Then

“Low latency acoustic modeling using temporal convolution and lstms,”
V. Peddinti, Y. Wang, D. Povey, and S. Khudanpur, · 2018
Later among the works it cites.
“Semi-orthogonal low-rank matrix factorization for deep neural networks,”
D. Povey, G. Cheng, Y. Wang, K. Li, H. Xu, M. Yarmohamadi, and S. Khudanpur, · 2018
Later among the works it cites.
“End-to-end speech recognition using lattice-free mmi,”
H. Hadian, H. Sameti, D. Povey, and S. Khudanpur, · 2018
Later among the works it cites.
“Phonetic and graphemic systems for multi-genre broadcast transcription,”
Y. Wang, X. Chen, M. Gales, A. Ragni, and J. Wong, · 2018
Later among the works it cites.
“State-of-the-art speech recognition with sequence-to-sequence models,”
C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina, N. Jaitly, B. Li, J. Chorowski, and M. Bacchiani, · 2018
Later among the works it cites.
“Improved training of end-to-end attention models for speech recognition,”
A. Zeyer, K. Irie, R. Schlüter, and H. Ney, · 2018
Later among the works it cites.
“No need for a lexicon? evaluating the value of the pronunciation lexica in end-to-end models,”
T. Sainath, R. Prabhavalkar, S. Kumar, S. Lee, A. Kannan, D. Rybach, V. Schogol, P. Nguyen, B. Li, Y. Wu, et al., · 2018
Later among the works it cites.
“Building competitive direct acoustics-to-word models for english conversational speech recognition,”
K. Audhkhasi, B. Kingsbury, B. Ramabhadran, G. Saon, and M. Picheny, · 2018
Later among the works it cites.
“Advancing acoustic-to-word ctc model,”
J. Li, G. Ye, A. Das, R. Zhao, and Y. Gong, · 2018
Later among the works it cites.
“Acoustic-to-word attention-based model complemented with character-level ctc-based model,”
S. Ueno, H. Inaguma, M. Mimura, and T. Kawahara, · 2018
Later among the works it cites.
“The CAPIO 2017 conversational speech recognition system,”
Original
Kyu J. Han, Akshay Chandrashekaran, Jungsuk Kim, and Ian R. Lane, · 2018
Later among the works it cites.
“Streaming end-to-end speech recognition for mobile devices,”
Y. He, T. N. Sainath, R. Prabhavalkar, I. McGraw, R. Alvarez, D. Zhao, D. Rybach, A. Kannan, Y. Wu, R. Pang, Q. Liang, D. Bhatia, Y. Shangguan, B. Li, G. Pundak, K. C. Sim, T. Bagby, S. Chang, K. Rao, and A. Gruenstein, · 2019
Closest in time.
“Specaugment: A simple data augmentation method for automatic speech recognition,”
D. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. Cubuk, and Q. Le, · 2019
Closest in time.
“RWTH ASR Systems for LibriSpeech: Hybrid vs Attention,”
C. Lüscher, E. Beck, K. Irie, M. Kitza, W. Michel, A. Zeyer, R. Schlüter, and H. Ney, · 2019
Closest in time.
“Sequence-to-sequence speech recognition with time-depth separable convolutions,”
A. Hannun, A. Lee, Q. Xu, and R. Collobert, · 2019
Closest in time.
“Jasper: An end-to-end convolutional neural acoustic model,”
J. Li, V. Lavrukhin, B. Ginsburg, R. Leary, O. Kuchaiev, J. M. Cohen, H. Nguyen, and R. T. Gadde, · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…