4-bit Quantization of LSTM-based Speech Recognition Models

J. J. Godfrey, E. C. Holliman, and J. McDaniel, “Switchboard: Telephone speech corpus for research and development,” in 1992 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , vol. 1, 1992, pp. 517–520

1992

Earlier work this paper cites.

F. Seide, G. Li, and D. Yu, “Conversational speech transcription using context-dependent deep neural networks,” in Proc. Interspeech , 2011, pp. 437–440

2011

Earlier work this paper cites.

T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, “Low-rank matrix factorization for deep neural network training with high-dimensional output targets,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2013, pp. 6655–6659

2013

Earlier work this paper cites.

Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432 , 2013

Original

2013

Earlier work this paper cites.

T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2015, pp. 3586–3589

2015

Earlier work this paper cites.

J. H. Wong and M. J. Gales, “Sequence student-teacher training of deep neural networks,” in Proc. Interspeech , 2016, pp. 2761–2765

2016

Earlier work this paper cites.

Y. Chebotar and A. Waters, “Distilling knowledge from ensembles of neural networks for speech recognition.” in Proc. Interspeech , 2016, pp. 3439–3443

2016

Earlier work this paper cites.

Q. He, H. Wen, S. Zhou, Y. Wu, C. Yao, X. Zhou, and Y. Zou, “Effective quantization methods for recurrent neural networks,” arXiv preprint arXiv:1611.10176 , 2016

Original

2016

Earlier work this paper cites.

G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L.-L. Lim, B. Roomi, and P. Hall, “English conversational telephone speech recognition by humans and machines,” in Proc. Interspeech , 2017, pp. 132–136

2017

Earlier work this paper cites.

I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Quantized neural networks: Training neural networks with low precision weights and activations,” The Journal of Machine Learning Research , vol. 18, no. 1, pp. 6869–6898, 2017

2017

Earlier work this paper cites.

X. Cui, V. Goel, and G. Saon, “Embedding-based speaker adaptive training of deep neural networks,” in Proc. Interspeech , 2017, pp. 122–126

2017

Earlier work this paper cites.

P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,” arXiv preprint arXiv:1706.02677 , 2017

Original

2017

Cited alongside, same era.

C. Xu, J. Yao, Z. Lin, W. Ou, Y. Cao, Z. Wang, and H. Zha, “Alternating multi-bit quantization for recurrent neural networks,” in International Conference on Learning Representations , 2018

2018

Cited alongside, same era.

J. Choi, P. I.-J. Chuang, Z. Wang, S. Venkataramani, V. Srinivasan, and K. Gopalakrishnan, “Bridging the accuracy gap for 2-bit quantized neural networks (qnn),” arXiv preprint arXiv:1807.06964 , 2018

Original

2018

Cited alongside, same era.

R. Krishnamoorthi, “Quantizing deep convolutional networks for efficient inference: A whitepaper,” arXiv preprint arXiv:1806.08342 , 2018

Original

2018

Cited alongside, same era.

J. Li, R. Zhao, H. Hu, and Y. Gong, “Improving rnn transducer modeling for end-to-end speech recognition,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) . IEEE, 2019, pp. 114–121

2019

Later among the works it cites.

Y. You, J. Hseu, C. Ying, J. Demmel, K. Keutzer, and C.-J. Hsieh, “Large-batch training for lstm and beyond,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , 2019, pp. 1–16

2019

Later among the works it cites.

S. Venkataramani, J. Choi, V. Srinivasan, W. Wang, J. Zhang, M. Schaal, M. J. Serrano, K. Ishizaki, H. Inoue, E. Ogawa, M. Ohara, L. Chang, and K. Gopalakrishnan, “Deeptools: Compiler and execution runtime extensions for rapid ai accelerator,” IEEE Micro , vol. 39, no. 5, pp. 102–111, 2019

2019

Later among the works it cites.

S. Venkataramani, V. Srinivasan, J. Choi, P. Heidelberger, L. Chang, and K. Gopalakrishnan, “Memory and interconnect optimizations for peta-scale deep learning systems,” in 2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC) , 2019, pp. 225–234