Built on

J. Salazar, K. Kirchhoff, and Z. Huang, “Self-attention networks for connectionist temporal classification in speech recognition,” 2019. [Online]. Available: https://arxiv.org/abs/1901.10055
Original
1901
Earlier work this paper cites.
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in
2006
Earlier work this paper cites.
T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and S. Khudanpur, “Recurrent neural network based language model,” in
2010
Earlier work this paper cites.
G. Hinton, L. Deng, D. Yu, G. Dahl, A. rahman Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition,”
2012
Earlier work this paper cites.
M. D. Zeiler, “ADADELTA: an adaptive learning rate method,” 2012. [Online]. Available: http://arxiv.org/abs/1212.5701
Original
2012
Earlier work this paper cites.
A. Graves, N. Jaitly, and A. rahman Mohamed, “Hybrid speech recognition with deep bidirectional lstm,” in
2013
Earlier work this paper cites.
A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in
2014
Earlier work this paper cites.
A. Y. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y. Ng, “Deep speech: Scaling up end-to-end speech recognition,” 2014. [Online]. Available: http://arxiv.org/abs/1412.5567
Original
2014
Earlier work this paper cites.
K. Cho, B. van Merrienboer, Ç. Gülçehre, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” 2014. [Online]. Available: http://arxiv.org/abs/1406.1078
Original
2014
Earlier work this paper cites.

Similar

I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” 2014. [Online]. Available: http://arxiv.org/abs/1409.3215
Original
2014
Cited alongside, same era.
J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” 2015. [Online]. Available: http://arxiv.org/abs/1506.07503
Original
2015
Cited alongside, same era.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” 2015
2015
Cited alongside, same era.
R. Collobert, C. Puhrsch, and G. Synnaeve, “Wav2letter: an end-to-end convnet-based speech recognition system,” 2016. [Online]. Available: http://arxiv.org/abs/1609.03193
Original
2016
Cited alongside, same era.

Then

S. Zhou, L. Dong, S. Xu, and B. Xu, “Syllable-based sequence-to-sequence speech recognition with the transformer in mandarin chinese,” in
Original
2018
Later among the works it cites.
D. Povey, H. Hadian, P. Ghahremani, K. Li, and S. Khudanpur, “A time-restricted self-attention layer for asr,”
2018
Later among the works it cites.
S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, “Espnet: End-to-end speech processing toolkit,” in
2018
Later among the works it cites.
T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,”
2018
Later among the works it cites.
K. J. Han, A. Chandrashekaran, J. Kim, and I. R. Lane, “The CAPIO 2017 conversational speech recognition system,” 2018. [Online]. Available: http://arxiv.org/abs/1801.00059

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2019

Transformers with convolutional context for ASR

Mohamed, Abdelrahman, Okhonko, Dmytro, Zettlemoyer, Luke

Understand

The recent success of transformer networks for neural machine translation and other NLP tasks has led to a surge in research work trying to apply it for speech recognition.

Recent efforts studied key research questions around ways of combining positional embedding with speech features, and stability of optimization for large scale learning of transformer networks.
In this paper, we propose replacing the sinusoidal positional embedding for transformers with convolutionally learned input representations.
These contextual representations provide subsequent transformer blocks with relative positional information needed for discovering long-range relationships between local concepts.

Built on

J. Salazar, K. Kirchhoff, and Z. Huang, “Self-attention networks for connectionist temporal classification in speech recognition,” 2019. [Online]. Available: https://arxiv.org/abs/1901.10055
Original
1901
Earlier work this paper cites.
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in
2006
Earlier work this paper cites.
T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and S. Khudanpur, “Recurrent neural network based language model,” in
2010
Earlier work this paper cites.
G. Hinton, L. Deng, D. Yu, G. Dahl, A. rahman Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition,”
2012
Earlier work this paper cites.
M. D. Zeiler, “ADADELTA: an adaptive learning rate method,” 2012. [Online]. Available: http://arxiv.org/abs/1212.5701
Original
2012
Earlier work this paper cites.
A. Graves, N. Jaitly, and A. rahman Mohamed, “Hybrid speech recognition with deep bidirectional lstm,” in
2013
Earlier work this paper cites.
A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in
2014
Earlier work this paper cites.
A. Y. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y. Ng, “Deep speech: Scaling up end-to-end speech recognition,” 2014. [Online]. Available: http://arxiv.org/abs/1412.5567
Original
2014
Earlier work this paper cites.
K. Cho, B. van Merrienboer, Ç. Gülçehre, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” 2014. [Online]. Available: http://arxiv.org/abs/1406.1078
Original
2014
Earlier work this paper cites.

Similar

I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” 2014. [Online]. Available: http://arxiv.org/abs/1409.3215
Original
2014
Cited alongside, same era.
J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” 2015. [Online]. Available: http://arxiv.org/abs/1506.07503
Original
2015
Cited alongside, same era.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” 2015
2015
Cited alongside, same era.
R. Collobert, C. Puhrsch, and G. Synnaeve, “Wav2letter: an end-to-end convnet-based speech recognition system,” 2016. [Online]. Available: http://arxiv.org/abs/1609.03193
Original
2016
Cited alongside, same era.

Then

S. Zhou, L. Dong, S. Xu, and B. Xu, “Syllable-based sequence-to-sequence speech recognition with the transformer in mandarin chinese,” in
Original
2018
Later among the works it cites.
D. Povey, H. Hadian, P. Ghahremani, K. Li, and S. Khudanpur, “A time-restricted self-attention layer for asr,”
2018
Later among the works it cites.
S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, “Espnet: End-to-end speech processing toolkit,” in
2018
Later among the works it cites.
T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,”
2018
Later among the works it cites.
K. J. Han, A. Chandrashekaran, J. Kim, and I. R. Lane, “The CAPIO 2017 conversational speech recognition system,” 2018. [Online]. Available: http://arxiv.org/abs/1801.00059

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in

2016

Cited alongside, same era.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017. [Online]. Available: http://arxiv.org/abs/1706.03762

Original

2017

Cited alongside, same era.

V. Liptchinsky, G. Synnaeve, and R. Collobert, “Letter-based speech recognition with gated convnets,”

Original

2017

Cited alongside, same era.

J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” 2018. [Online]. Available: http://arxiv.org/abs/1810.04805

Original

2018

Cited alongside, same era.

M. Sperber, J. Niehues, G. Neubig, S. Stüker, and A. Waibel, “Self-attentional acoustic models,” 2018. [Online]. Available: http://arxiv.org/abs/1803.09519

Original

2018

Cited alongside, same era.

Original

2018

Later among the works it cites.

A. Zeyer, K. Irie, R. Schlüter, and H. Ney, “Improved training of end-to-end attention models for speech recognition,” 2018. [Online]. Available: http://arxiv.org/abs/1805.03294

Original

2018

Later among the works it cites.

N. Zeghidour, Q. Xu, V. Liptchinsky, N. Usunier, G. Synnaeve, and R. Collobert, “Fully convolutional speech recognition,”

Original

2018

Later among the works it cites.

A. Hannun, A. Lee, Q. Xu, and R. Collobert, “Sequence-to-sequence speech recognition with time-depth separable convolutions,” 2019. [Online]. Available: Under Review

2019

Closest in time.

S. Sabour, W. Chan, and M. Norouzi, “Optimal completion distillation for sequence learning,” in

2019

Closest in time.