Fetching the paper…

Reading the bibliography…

2019

Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion

Sun, Hao, Tan, Xu, Gan, Jun-Wei et al.

Original
PDF
Post on X

Understand

Grapheme-to-phoneme (G2P) conversion is an important task in automatic speech recognition and text-to-speech systems.

Recently, G2P conversion is viewed as a sequence to sequence task and modeled by RNN or CNN based encoder-decoder framework.
However, previous works do not consider the practical issues when deploying G2P model in the production system, such as how to leverage additional unlabeled data to boost the accuracy, as well as reduce model size for online deployment.
In this work, we propose token-level ensemble distillation for G2P conversion, which can (1) boost the accuracy by distilling the knowledge from additional unlabeled data, and (2) reduce the model size but maintain the high accuracy, both of which are very practical and helpful in the online production system.

Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion · Around

Built on

T. G. Dietterich, “Ensemble methods in machine learning,” in
2000
Earlier work this paper cites.
S. F. Chen, “Conditional and joint models for grapheme-to-phoneme conversion,” in
2003
Earlier work this paper cites.
C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, “Model compression,” in
2006
Earlier work this paper cites.
M. Bisani and H. Ney, “Joint-sequence models for grapheme-to-phoneme conversion,”
2008
Earlier work this paper cites.
K. Wu, C. Allauzen, K. Hall, M. Riley, and B. Roark, “Encoding linear models as weighted finite-state transducers,” in
2014
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,”
Original
2014
Earlier work this paper cites.
K. Yao and G. Zweig, “Sequence-to-sequence neural net models for grapheme-to-phoneme conversion,”
Original
2015
Earlier work this paper cites.
G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”
Original
2015
Earlier work this paper cites.
T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” in
2015
Earlier work this paper cites.
K. Rao, F. Peng, H. Sak, and F. Beaufays, “Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks,” in
2015
Earlier work this paper cites.
Y. Kim and A. M. Rush, “Sequence-level knowledge distillation,”
Original
2016
Earlier work this paper cites.

Similar

Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey
Original
2016
Cited alongside, same era.
S. Toshniwal and K. Livescu, “Jointly learning to align and convert graphemes to phonemes with neural attention models,” in
2016
Cited alongside, same era.
M. Freitag, Y. Al-Onaizan, and B. Sankaran, “Ensemble distillation for neural machine translation,”
Original
2017
Cited alongside, same era.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in
2017
Cited alongside, same era.
B. Milde, C. Schmidt, and J. Köhler, “Multitask sequence-to-sequence models for grapheme-to-phoneme conversion.” in
2017
Cited alongside, same era.

Then

J. Guo, X. Tan, D. He, T. Qin, L. Xu, and T.-Y. Liu, “Non-autoregressive neural machine translation with enhanced decoder input,”
Original
2018
Later among the works it cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”
Original
2018
Later among the works it cites.
M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser, “Universal transformers,”
Original
2018
Later among the works it cites.
A. Wangperawong, “Attending to mathematical language with transformers,”
Original
2018
Later among the works it cites.
R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton, “Large scale distributed neural network training through online distillation,”

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin, “Convolutional sequence to sequence learning,” in

2017

Cited alongside, same era.

M.-j. Chae, K. Park, L. Bang, S. Suh, L. Park, N. Kimt, and L. Park, “Convolutional sequence to sequence model with non-sequential greedy decoding for grapheme to phoneme conversion,” in

2018

Cited alongside, same era.

T. Furlanello, Z. C. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, “Born again neural networks,”

2018

Cited alongside, same era.

C. Gong, X. Tan, D. He, and T. Qin, “Sentence-wise smooth regularization for sequence to sequence learning,” in

2018

Cited alongside, same era.

S. Zhou, L. Dong, S. Xu, and B. Xu, “A comparison of modeling units in sequence-to-sequence speech recognition with the transformer on mandarin chinese,” in

2018

Cited alongside, same era.

H. Hassan, A. Aue, C. Chen, V. Chowdhary, J. Clark, C. Federmann, X. Huang, M. Junczys-Dowmunt, W. Lewis, M. Li

2018

Cited alongside, same era.

2018

Later among the works it cites.

C. Yang, L. Xie, S. Qiao, and A. Yuille, “Knowledge distillation in generations: More tolerant teachers educate better students,”

2018

Later among the works it cites.

Y. Ren, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Almost unsupervised text to speech and automatic speech recognition,”

2019

Closest in time.

Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech: Fast, robust and controllable text to speech,”

2019

Closest in time.

X. Tan, Y. Ren, D. He, T. Qin, and T.-Y. Liu, “Multilingual neural machine translation with knowledge distillation,” in

2019

Closest in time.

D. R. So, C. Liang, and Q. V. Le, “The evolved transformer,”

2019

Closest in time.

K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu, “Mass: Masked sequence to sequence pre-training for language generation,”

2019

Closest in time.