Fetching the paper…

Reading the bibliography…

2020

Aligned Cross Entropy for Non-Autoregressive Machine Translation

Ghazvininejad, Marjan, Karpukhin, Vladimir, Zettlemoyer, Luke et al.

Original
PDF
Post on X

Understand

Non-autoregressive machine translation models significantly speed up decoding by allowing for parallel prediction of the entire target sequence.

However, modeling word order is more challenging due to the lack of autoregressive factors in the model.
This difficultly is compounded during training with cross entropy loss, which can highly penalize small shifts in word order.
In this paper, we propose aligned cross entropy (AXE) as an alternative loss function for training of non-autoregressive models.

Aligned Cross Entropy for Non-Autoregressive Machine Translation · Around

Built on

Insertion transformer: Flexible sequence generation via insertion operations
Original
Stern, M., Chan, W., Kiros, J. R., and Uszkoreit, J · 1902
Earlier work this paper cites.
Non-autoregressive machine translation with auxiliary regularization
Original
Wang, Y., Tian, F., He, D., Qin, T., Zhai, C., and Liu, T.-Y · 1902
Earlier work this paper cites.
compare-mt: A tool for holistic comparison of language generation systems
Original
Neubig, G., Dou, Z.-Y., Hu, J., Michel, P., Pruthi, D., and Wang, X · 1903
Earlier work this paper cites.
Levenshtein transformer
Original
Gu, J., Wang, C., and Zhao, J · 1905
Earlier work this paper cites.
Dynamic programming algorithm optimization for spoken word recognition
Sakoe, H. and Chiba, S · 1978
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J · 2002
Earlier work this paper cites.
Understanding knowledge distillation in non-autoregressive machine translation
Original
Zhou, C., Neubig, G., and Gu, J · 2004
Earlier work this paper cites.
Curriculum learning
Bengio, Y., Louradour, J., Collobert, R., and Weston, J · 2009
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.

Similar

Sequence-level knowledge distillation
Original
Kim, Y. and Rush, A. M · 2016
Cited alongside, same era.
Neural machine translation of rare words with subword units
Sennrich, R., Haddow, B., and Birch, A · 2016
Cited alongside, same era.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Cited alongside, same era.
Non-autoregressive neural machine translation
Gu, J., Bradbury, J., Xiong, C., Li, V. O., and Socher, R · 2018
Cited alongside, same era.
Deterministic non-autoregressive neural sequence modeling by iterative refinement

Then

Hint-based training for non-autoregressive machine translation
Original
Li, Z., Lin, Z., He, D., Tian, F., Qin, T., Wang, L., and Liu, T.-Y · 2019
Later among the works it cites.
Flowseq: Non-autoregressive conditional sequence generation with generative flow
Original
Ma, X., Zhou, C., Li, X., Neubig, G., and Hovy, E · 2019
Later among the works it cites.
Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation
Original
Shao, C., Zhang, J., Feng, Y., Meng, F., and Zhou, J · 2019
Later among the works it cites.
Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior
Original
Shu, R., Lee, J., Nakayama, H., and Cho, K · 2019
Later among the works it cites.
Fast structured decoding for sequence models

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

Lee, J. D., Mansimov, E., and Cho, K · 2018

Cited alongside, same era.

End-to-end non-autoregressive neural machine translation with connectionist temporal classification

Libovický, J. and Helcl, J · 2018

Cited alongside, same era.

A call for clarity in reporting BLEU scores

Post, M · 2018

Cited alongside, same era.

Mask-predict: Parallel decoding of conditional masked language models

Ghazvininejad, M., Levy, O., Liu, Y., and Zettlemoyer, L · 2019

Cited alongside, same era.

Sun, Z., Li, Z., Wang, H., He, D., Lin, Z., and Deng, Z · 2019

Later among the works it cites.

Pay less attention with lightweight and dynamic convolutions

Wu, F., Fan, A., Baevski, A., Dauphin, Y. N., and Auli, M · 2019

Later among the works it cites.

Non-autoregressive video captioning with iterative refinement

Yang, B., Liu, F., and Zou, Y · 2019

Later among the works it cites.

Semi-autoregressive training improves mask-predict decoding

Ghazvininejad, M., Levy, O., and Zettlemoyer, L · 2020

Closest in time.

Parallel machine translation with disentangled context transformer

Kasai, J., Cross, J., Ghazvininejad, M., and Gu, J · 2020

Closest in time.