2016

Achieving Human Parity in Conversational Speech Recognition

Xiong, W., Droppo, J., Huang, X. et al.

Understand

Conversational speech recognition has served as a flagship speech recognition task since the release of the Switchboard corpus in the 1990s.

In this paper, we measure the human error rate on the widely used NIST 2000 test set, and find that our latest automated system has reached human parity.
The error rate of professional transcribers is 5.9% for the Switchboard portion of the data, in which newly acquainted pairs of people discuss an assigned topic, and 11.3% for the CallHome portion where friends and family members have open-ended conversations.
In both cases, our automated system establishes a new state of the art, and edges past the human benchmark, achieving error rates of 5.8% and 11.0%, respectively.

Built on

“Generalization of back-propagation to recurrent neural networks”,
F. J. Pineda, · 1987
Earlier work this paper cites.
“The DARPA 1000-word resource management database for continuous speech recognition”,
P. Price, W. M. Fisher, J. Bernstein, and D. S. Pallett, · 1988
Earlier work this paper cites.
“A learning algorithm for continually running fully recurrent neural networks”,
R. J. Williams and D. Zipser, · 1989
Earlier work this paper cites.
“Phoneme recognition using time-delay neural networks”,
A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang, · 1989
Earlier work this paper cites.
“Backpropagation applied to handwritten zip code recognition”,
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, · 1989
Earlier work this paper cites.
“Consonant recognition by modular construction of large phonemic time-delay neural networks”,
A. Waibel, H. Sawai, and K. Shikano, · 1989
Earlier work this paper cites.
“A recurrent error propagation network speech recognition system”,
T. Robinson and F. Fallside, · 1991
Earlier work this paper cites.
“The design for the wall street journal-based csr corpus”,
D. B. Paul and J. M. Baker, · 1992
Earlier work this paper cites.
“Switchboard: Telephone speech corpus for research and development”,
J. J. Godfrey, E. C. Holliman, and J. McDaniel, · 1992
Earlier work this paper cites.
“Convolutional networks for images, speech, and time series”,
Y. LeCun and Y. Bengio, · 1995
Earlier work this paper cites.
“The 1996 broadcast news speech and language-model corpus”,
D. Graff, Z. Wu, R. MacIntyre, and M. Liberman, · 1997
Earlier work this paper cites.
“Speech recognition by machines and humans”,
R. P. Lippmann, · 1997
Earlier work this paper cites.
“Long short-term memory”,
S. Hochreiter and J. Schmidhuber, · 1997
Earlier work this paper cites.
“The SRI March 2000 Hub-5 conversational speech transcription system”,
A. Stolcke et al., · 2000
Earlier work this paper cites.
“The AT&T 2001 LVCSR system”,
A. Ljolje, · 2001
Earlier work this paper cites.
“Deep Blue”,
M. Campbell, A. J. Hoane, and F.-h. Hsu, · 2002
Earlier work this paper cites.
“SRILM—an extensible language modeling toolkit”,
A. Stolcke, · 2002
Earlier work this paper cites.
“A look at NIST’s benchmark ASR tests: past, present, and future”,
D. S. Pallett, · 2003
Earlier work this paper cites.
“Conversational telephone speech recognition”,
J.-L. Gauvain, L. Lamel, H. Schwenk, G. Adda, L. Chen, and F. Lefevre, · 2003
Earlier work this paper cites.
“The Fisher corpus: a resource for the next generations of speech-to-text”,
C. Cieri, D. Miller, and K. Walker, · 2004
Earlier work this paper cites.
“Development of the 2003 cu-htk conversational telephone speech transcription system”,
G. Evermann, H. Y. Chan, M. J. F. Gales, T. Hain, X. Liu, D. Mrva, L. Wang, and P. C. Woodland, · 2004
Earlier work this paper cites.
“Framewise phoneme classification with bidirectional LSTM and other neural network architectures”,
A. Graves and J. Schmidhuber, · 2005
Earlier work this paper cites.
“Super-human multi-talker speech recognition: the IBM 2006 Speech Separation Challenge system”,
T. T. Kristjansson, J. R. Hershey, P. A. Olsen, S. J. Rennie, and R. A. Gopinath, · 2006
Earlier work this paper cites.

Similar

“Advances in speech transcription at IBM under the DARPA EARS program”,
S. F. Chen, B. Kingsbury, L. Mangu, D. Povey, G. Saon, H. Soltau, and G. Zweig, · 2006
Cited alongside, same era.
“Advances in transcription of broadcast news and conversational telephone speech within the combined ears bbn/limsi system”,
S. Matsoukas, J.-L. Gauvain, G. Adda, T. Colthurst, C.-L. Kao, O. Kimball, L. Lamel, F. Lefevre, J. Z. Ma, J. Makhoul, et al., · 2006
Cited alongside, same era.
“Recent innovations in speech-to-text transcription at SRI-ICSI-UW”,
A. Stolcke, B. Chen, H. Franco, V. R. R. Gadde, M. Graciarena, M.-Y. Hwang, K. Kirchhoff, A. Mandal, N. Morgan, X. Lei, et al., · 2006
Cited alongside, same era.
“Recurrent neural network based language model”,
T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur, · 2010
Cited alongside, same era.
“Transcription methods for consistency, volume and efficiency”,
M. L. Glenn, S. Strassel, H. Lee, K. Maeda, R. Zakhary, and X. Li, · 2010
Cited alongside, same era.
“Noise-contrastive estimation: A new estimation principle for unnormalized statistical models”,
M. Gutmann and A. Hyvärinen, · 2010
Cited alongside, same era.
“Conversational speech transcription using context-dependent deep neural networks”,
F. Seide, G. Li, and D. Yu, · 2011
Cited alongside, same era.
“Front-end factor analysis for speaker verification”,
N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, · 2011
Cited alongside, same era.
“Sequential classification criteria for NNs in automatic speech recognition”,
G. Wang and K. Sim, · 2011
Cited alongside, same era.
“Context dependent recurrent neural network language model”,
T. Mikolov and G. Zweig, · 2012
Cited alongside, same era.
“Lstm neural networks for language modeling.”,
M. Sundermeyer, R. Schlüter, and H. Ney, · 2012
Cited alongside, same era.
“Efficient estimation of maximum entropy language models with N-gram features: An SRILM extension”,
T. Alumäe and M. Kurimo, · 2012
Cited alongside, same era.

Then

“An introduction to computational networks and the Computational Network Toolkit”,
D. Yu et al., · 2014
Later among the works it cites.
“1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs”,
F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, · 2014
Later among the works it cites.
“Deep Speech 2: End-to-end speech recognition in English and Mandarin”,
Original
D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, et al., · 2015
Later among the works it cites.
“Fast and accurate recurrent neural network acoustic models for speech recognition”,
H. Sak, A. Senior, K. Rao, and F. Beaufays, · 2015
Later among the works it cites.
“The IBM 2015 English conversational telephone speech recognition system”,
G. Saon, H.-K. J. Kuo, S. Rennie, and M. Picheny, · 2015
Later among the works it cites.
“Very deep convolutional neural networks for LVCSR”,
M. Bi, Y. Qian, and K. Yu, · 2015
Later among the works it cites.
“Deep residual learning for image recognition”,
Original
K. He, X. Zhang, S. Ren, and J. Sun, · 2015
Later among the works it cites.
“Highway networks”,
Original
R. K. Srivastava, K. Greff, and J. Schmidhuber, · 2015
Later among the works it cites.
“Mastering the game of Go with deep neural networks and tree search”,
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al., · 2016
Closest in time.
“Very deep multilingual convolutional neural networks for LVCSR”,
T. Sercu, C. Puhrsch, B. Kingsbury, and Y. LeCun, · 2016
Closest in time.
“Very deep convolutional neural networks for noise robust speech recognition”,
Y. Qian, M. Bi, T. Tan, and K. Yu, · 2016
Closest in time.
“Improving English conversational telephone speech recognition”,
I. Medennikov, A. Prudnikov, and A. Zatvornitskiy, · 2016
Closest in time.
“Linearly augmented deep neural network”,
P. Ghahremani, J. Droppo, and M. L. Seltzer, · 2016
Closest in time.
“Deep convolutional neural networks with layer-wise context expansion and attention”,
D. Yu, W. Xiong, J. Droppo, A. Stolcke, G. Ye, J. Li, and G. Zweig, · 2016
Closest in time.
“The IBM 2016 English conversational telephone speech recognition system”,
G. Saon, T. Sercu, S. J. Rennie, and H. J. Kuo, · 2016
Closest in time.
“Purely sequence-trained neural networks for ASR based on lattice-free MMI”,
D. Povey, V. Peddinti, D. Galvez, P. Ghahrmani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, · 2016
Closest in time.
“Parallelizing WFST speech decoders”,
C. Mendis, J. Droppo, S. Maleki, M. Musuvathi, T. Mytkowicz, and G. Zweig, · 2016
Closest in time.
“CUED-RNNLM: An open-source toolkit for efficient training and evaluation of recurrent neural network language models”,
X. Chen, X. Liu, Y. Qian, M. J. F. Gales, and P. C. Woodland, · 2016
Closest in time.
“Using the output embedding to improve language models”,
Original
O. Press and L. Wolf, · 2016
Closest in time.
“Self-stabilized deep neural network”,
P. Ghahremani and J. Droppo, · 2016
Closest in time.
“Stimulated deep neural network for speech recognition”,
C. Wu, P. Karanasou, M. J. Gales, and K. C. Sim, · 2016
Closest in time.
“The Microsoft 2016 conversational speech recognition system”,
Original
W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig, · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…