2020

Imputer: Sequence Modelling via Imputation and Dynamic Programming

Chan, William, Saharia, Chitwan, Hinton, Geoffrey et al.

Understand

This paper presents the Imputer, a neural sequence model that generates output sequences iteratively via imputations.

The Imputer is an iterative generative model, requiring only a constant number of generation steps independent of the number of input or output tokens.
The Imputer can be trained to approximately marginalize over all possible alignments between the input and output sequences, and all possible generation orders.
We present a tractable dynamic programming training algorithm, which yields a lower bound on the log marginal likelihood.

Built on

The Design for the Wall Street Journal-based CSR Corpus
Paul, D. and Baker, J · 1992
Earlier work this paper cites.
Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks
Graves, A., Fernandez, S., Gomez, F., and Schmidhuber, J · 2006
Earlier work this paper cites.
The Kaldi Speech Recognition Toolkit
Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlicek, P., Qian, Y., Schwarz, P., Silovsky, J., Stemmer, G., and Vesely, K · 2011
Earlier work this paper cites.
Sequence Transduction with Recurrent Neural Networks
Graves, A · 2012
Earlier work this paper cites.
Speech Recognition with Deep Recurrent Neural Networks
Graves, A., Mohamed, A., and Hinton, G · 2013
Earlier work this paper cites.
Deep Convolutional Neural Networks for LVCSR
Sainath, T., Mohamed, A., Kingsbury, B., and Ramabhadran, B · 2013
Earlier work this paper cites.
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y · 2014
Earlier work this paper cites.
Towards End-to-End Speech Recognition with Recurrent Neural Networks
Graves, A. and Jaitly, N · 2014
Earlier work this paper cites.
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., and Le, Q · 2014
Earlier work this paper cites.
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., and Bengio, Y · 2015
Earlier work this paper cites.
Effective Approaches to Attention-based Neural Machine Translation
Luong, M.-T., Pham, H., and Manning, C. D · 2015
Earlier work this paper cites.
Librispeech: An ASR corpus based on public domain audio books
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S · 2015
Earlier work this paper cites.
Show and Tell: A Neural Image Caption Generator
Vinyals, O., Toshev, A., Bengio, S., and Erhan, D · 2015
Earlier work this paper cites.
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., and Bengio, Y · 2015
Earlier work this paper cites.
Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition
Chan, W., Jaitly, N., Le, Q., and Vinyals, O · 2016
Earlier work this paper cites.
Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
Collobert, R., Puhrsch, C., and Synnaeve, G · 2016
Earlier work this paper cites.
Learning Online Alignments with Continuous Rewards Policy Gradient
Luo, Y., Chiu, C.-C., Jaitly, N., and Sutskever, I · 2016
Earlier work this paper cites.

Similar

WaveNet: A Generative Model for Raw Audio
Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K · 2016
Cited alongside, same era.
Exploring Neural Transducers for End-to-End Speech Recognition
Battenberg, E., Chen, J., Child, R., Coates, A., Gaur, Y., Li, Y., Liu, H., Satheesh, S., Seetapun, D., Sriram, A., and Zhu, Z · 2017
Cited alongside, same era.
Latent Sequence Decompositions
Chan, W., Zhang, Y., and Jaitly, N · 2017
Cited alongside, same era.
Towards better decoding and language model integration in sequence to sequence models
Chorowski, J. and Jaitly, N · 2017
Cited alongside, same era.
Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning
Kim, S., Hori, T., and Watanabe, S · 2017
Cited alongside, same era.
Letter-Based Speech Recognition with Gated ConvNets
Liptchinsky, V., Synnaeve, G., and Collobert, R · 2017
Cited alongside, same era.
Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling
Liu, H., Zhu, Z., Li, X., and Satheesh, S · 2017
Cited alongside, same era.
A Comparison of Sequence-to-Sequence Models for Speech Recognition
Prabhavalkar, R., Rao, K., Sainath, T. N., Li, B., Johnson, L., and Jaitly, N · 2017
Cited alongside, same era.
Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017
Cited alongside, same era.
Very Deep Convolutional Networks for End-to-End Speech Recognition
Zhang, Y., Chan, W., and Jaitly, N · 2017
Cited alongside, same era.
State-of-the-art Speech Recognition With Sequence-to-Sequence Models
Chiu, C.-C., Sainath, T., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R. J., Rao, K., Gonina, E., Jaitly, N., Li, B., Chorowski, J., and Bacchiani, M · 2018
Cited alongside, same era.
Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition
Dong, L., Xu, S., and Xu, B · 2018
Cited alongside, same era.

Then

Sequence-to-Sequence ASR Optimization via Reinforcement Learning
Tjandra, A., Sakti, S., and Nakamura, S · 2018
Later among the works it cites.
Non-Autoregressive Transformer Automatic Speech Recognition
Chen, N., Watanabe, S., Villalba, J., and Dehak, N · 2019
Later among the works it cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2019
Later among the works it cites.
Mask-Predict: Parallel Decoding of Conditional Masked Language Models
Ghazvininejad, M., Levy, O., Liu, Y., and Zettlemoyer, L · 2019
Later among the works it cites.
Model Unit Exploration for Sequence-to-Sequence Speech Recognition
Irie, K., Prabhavalkar, R., Kannan, A., Bruguier, A., Rybach, D., and Nguyen, P · 2019
Later among the works it cites.
Jasper: An End-to-End Convolutional Neural Acoustic Model
Li, J., Lavrukhin, V., Ginsburg, B., Leary, R., Kuchaiev, O., Cohen, J. M., Nguyen, H., and Gadde, R. T · 2019
Later among the works it cites.
Big Bidirectional Insertion Representations for Documents
Li, L. and Chan, W · 2019
Later among the works it cites.
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention – w/o Data Augmentation
Luscher, C., Beck, E., Irie, K., Kitza, M., Michel, W., Zeyer, A., Schluter, R., and Ney, H · 2019
Later among the works it cites.
Transformers with convolutional context for ASR
Mohamed, A., Okhonko, D., and Zettlemoyer, L · 2019
Later among the works it cites.
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E., and Le, Q · 2019
Later among the works it cites.
Insertion-Deletion Transformer
Ruis, L., Stern, M., Proskurnia, J., and Chan, W · 2019
Later among the works it cites.
Optimal Completion Distillation for Sequence Learning
Sabour, S., Chan, W., and Norouzi, M · 2019
Later among the works it cites.
Insertion Transformer: Flexible Sequence Generation via Insertion Operations
Stern, M., Chan, W., Kiros, J., and Uszkoreit, J · 2019
Later among the works it cites.
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
Synnaeve, G., Xu, Q., Kahn, J., Grave, E., Likhomanenko, T., Pratap, V., Sriram, A., Liptchinsky, V., and Collobert, R · 2019
Later among the works it cites.
Advancing Sequence-to-Sequence Based Speech Recognition
Tuske, Z., Audhkasi, K., and Saon, G · 2019
Later among the works it cites.
Non-Monotonic Sequential Text Generation
Welleck, S., Brantley, K., Daume, H., and Cho, K · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…