2017

Kernel Approximation Methods for Speech Recognition

May, Avner, Garakani, Alireza Bagheri, Lu, Zhiyun et al.

Understand

We study large-scale kernel methods for acoustic modeling in speech recognition and compare their performance to deep neural networks (DNNs).

We perform experiments on four speech recognition datasets, including the TIMIT and Broadcast News benchmark tasks, and compare these two types of models on frame-level performance metrics (accuracy, cross-entropy), as well as on recognition metrics (word/character error rate).
In order to scale kernel methods to these large datasets, we use the random Fourier feature method of Rahimi and Recht (2007).
We propose two novel techniques for improving the performance of kernel acoustic models.

Built on

The application of hidden Markov models in speech recognition
Mark Gales and Steve Young · 1932
Earlier work this paper cites.
Maximum mutual information estimation of hidden Markov model parameters for speech recognition
L. Bahl, P. Brown, P. de Souza, and R. Mercer · 1986
Earlier work this paper cites.
Approximation by superpositions of a sigmoidal function
George Cybenko · 1989
Earlier work this paper cites.
Multilayer feedforward networks are universal approximators
Kurt Hornik, Maxwell B. Stinchcombe, and Halbert White · 1989
Earlier work this paper cites.
Generalization and parameter estimation in feedforward nets: Some experiments
N. Morgan and H. Bourlard · 1990
Earlier work this paper cites.
DARPA TIMIT acoustic phonetic continuous speech corpus CDROM, 1993
J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren · 1993
Earlier work this paper cites.
MMIE training of large vocabulary recognition systems
V Valtchev, J.J Odell, P.C Woodland, and S.J Young · 1997
Earlier work this paper cites.
Maximum likelihood linear transformations for HMM-based speech recognition
M.J.F. Gales · 1998
Earlier work this paper cites.
Fast Training of Support Vector Machines using Sequential Minimal Optimization
John C. Platt · 1998
Earlier work this paper cites.
A novel loss function for the overall risk criterion based discriminative training of HMM models
Janez Kaiser, Bogomir Horvat, and Zdravko Kacic · 2000
Earlier work this paper cites.
Using the Nyström method to speed up kernel machines
C.K.I. Williams and M. Seeger · 2001
Earlier work this paper cites.
Localized Rademacher complexities
Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson · 2002
Earlier work this paper cites.
Training Invariant Support Vector Machines
Dennis DeCoste and Bernhard Schölkopf · 2002
Earlier work this paper cites.
Minimum phone error and i-smoothing for improved discriminative training
D. Povey and P. C. Woodland · 2002
Earlier work this paper cites.
Learning with kernels
B. Schölkopf and A. Smola · 2002
Earlier work this paper cites.
2003 NIST Rich Transcription evaluation data
Jonathan Fiscus, George Doddington, Audrey Le, Greg Sanders, Mark Przybocki, and David Pallett · 2003
Earlier work this paper cites.
Sparseness of support vector machines—some asymptotically sharp bounds
I. Steinwart · 2004
Earlier work this paper cites.
Core Vector Machines: Fast SVM Training on Very Large Data Sets
Ivor W. Tsang, James T. Kwok, and Pak-Ming Cheung · 2005
Earlier work this paper cites.
Hypothesis spaces for minimum Bayes risk training in large vocabulary speech recognition
Matthew Gibson and Thomas Hain · 2006
Earlier work this paper cites.
A fast learning algorithm for deep belief nets
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh · 2006
Earlier work this paper cites.
Large Scale Kernel Machines
Léon Bottou, Olivier Chapelle, Dennis DeCoste, and Jason Weston, editors · 2007
Earlier work this paper cites.
Evaluation of proposed modifications to MPE for large scale discriminative training
Daniel Povey and Brian Kingsbury · 2007
Earlier work this paper cites.

Similar

Random features for large-scale kernel machines
Ali Rahimi and Benjamin Recht · 2007
Cited alongside, same era.
Boosted MMI for model and feature-space discriminative training
D. Povey, D. Kanevsky, B. Kingsbury, B. Ramabhadran, G. Saon, and K. Visweswariah · 2008
Cited alongside, same era.
Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning
Ali Rahimi and Benjamin Recht · 2008
Cited alongside, same era.
Efficient online and batch learning using forward backward splitting
John C. Duchi and Yoram Singer · 2009
Cited alongside, same era.
Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling
Brian Kingsbury · 2009
Cited alongside, same era.
Coresets, Sparse Greedy Approximation, and the Frank-Wolfe Algorithm
Kenneth L. Clarkson · 2010
Cited alongside, same era.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Cited alongside, same era.
The IBM attila speech recognition toolkit
Hagen Soltau, George Saon, and Brian Kingsbury · 2010
Cited alongside, same era.
COFFIN: A computational framework for linear svms
Sören Sonnenburg and Vojtech Franc · 2010
Cited alongside, same era.
Arccosine Kernels: Acoustic Modeling with Infinite Neural Networks
C.-C. Cheng and B. Kingsbury · 2011
Cited alongside, same era.
Making deep belief networks effective for large vocabulary continuous speech recognition
T.N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P. Novak, and A. Mohamed · 2011
Cited alongside, same era.
Feature engineering in context-dependent deep neural networks for conversational speech transcription
Frank Seide, Gang Li, Xie Chen, and Dong Yu · 2011
Cited alongside, same era.

Then

Low-rank Matrix Factorization for Deep Neural Network Training with High-dimensional Output Targets
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arısoy, and Bhuvana Ramabhadran · 2013
Later among the works it cites.
On the importance of initialization and momentum in deep learning
Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton · 2013
Later among the works it cites.
Sequence-discriminative training of deep neural networks
Karel Veselý, Arnab Ghoshal, Lukás Burget, and Daniel Povey · 2013
Later among the works it cites.
Convolutional neural networks for speech recognition
Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu · 2014
Later among the works it cites.
Do deep nets really need to be deep?
Jimmy Ba and Rich Caruana · 2014
Later among the works it cites.
On the complexity of neural network classifiers: A comparison between shallow and deep architectures
Monica Bianchini and Franco Scarselli · 2014
Later among the works it cites.
Scalable kernel methods via doubly stochastic gradients
Bo Dai, Bo Xie, Niao He, Yingyu Liang, Anant Raj, Maria-Florina Balcan, and Le Song · 2014
Later among the works it cites.
Compact random feature maps
Raffay Hamid, Ying Xiao, Alex Gittens, and Dennis DeCoste · 2014
Later among the works it cites.
Kernel methods match deep neural networks on TIMIT
Po-Sen Huang, Haim Avron, Tara N. Sainath, Vikas Sindhwani, and Bhuvana Ramabhadran · 2014
Later among the works it cites.
A simple proof of Stirling’s formula for the gamma function
G. J. O. Jameson · 2014
Later among the works it cites.
On the number of linear regions of deep neural networks
Guido F. Montúfar, Razvan Pascanu, KyungHyun Cho, and Yoshua Bengio · 2014
Later among the works it cites.
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Hasim Sak, Andrew W. Senior, and Françoise Beaufays · 2014
Later among the works it cites.
Personal communication, 2014
Alex Smola · 2014
Later among the works it cites.
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Later among the works it cites.
Sparse random feature algorithm as coordinate descent in Hilbert space
E.-H. Yen, T.-W. Lin, S.-D. Lin, P.K. Ravikumar, and I.S. Dhillon · 2014
Later among the works it cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy · 2015
Later among the works it cites.
Spherical random features for polynomial kernels
Jeffrey Pennington, Felix X. Yu, and Sanjiv Kumar · 2015
Later among the works it cites.
Compact nonlinear maps and circulant extensions
Original
Felix X. Yu, Sanjiv Kumar, Henry A. Rowley, and Shih-Fu Chang · 2015
Later among the works it cites.
A comparison between deep neural nets and kernel acoustic models for speech recognition
Zhiyun Lu, Dong Quo, Alireza Bagheri Garakani, Kuan Liu, Avner May, Aurélien Bellet, Linxi Fan, Michael Collins, Brian Kingsbury, Michael Picheny, and Fei Sha · 2016
Later among the works it cites.
Compact kernel models for acoustic modeling via random feature selection
Avner May, Michael Collins, Daniel J. Hsu, and Brian Kingsbury · 2016
Later among the works it cites.
Purely sequence-trained neural networks for asr based on lattice-free mmi
Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahrmani, Vimal Manohar, Xingyu Na, Yiming Wang, and Sanjeev Khudanpur · 2016
Later among the works it cites.
Achieving human parity in conversational speech recognition
Original
W. Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig · 2016
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…