2019

On NMT Search Errors and Model Errors: Cat Got Your Tongue?

Stahlberg, Felix, Byrne, Bill

Understand

We report on search errors and model errors in neural machine translation (NMT).

We present an exact inference procedure for neural sequence models based on a combination of beam search and depth-first search.
We use our exact search to find the global best model scores under a Transformer base model for the entire WMT15 English-German test set.
Surprisingly, beam search fails to find these global best model scores in most cases, even with a very large beam size of 100.

Calibration of encoder decoder models for neural machine translation
Original
Aviral Kumar and Sunita Sarawagi. 2019 · 1903
Earlier work this paper cites.
Audio chord recognition with recurrent neural networks
Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. 2013 · 2013
Earlier work this paper cites.
Recurrent continuous translation models
Nal Kalchbrenner and Phil Blunsom. 2013 · 2013
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014 · 2014
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015 · 2015
Earlier work this paper cites.
Montreal neural machine translation systems for WMT’15
Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015 · 2015
Earlier work this paper cites.
Improved neural machine translation with SMT features
Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016 · 2016
Earlier work this paper cites.

Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Cited alongside, same era.
Length bias in encoder decoder models and a case for global conditioning
Pavel Sountsov and Sunita Sarawagi. 2016 · 2016
Cited alongside, same era.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016 · 2016
Cited alongside, same era.
When to finish? Optimal beam search for neural text generation (modulo beam size)
Liang Huang, Kai Zhao, and Mingbo Ma. 2017 · 2017
Cited alongside, same era.
Depthwise separable convolutions for neural machine translation
Original
Lukasz Kaiser, Aidan N Gomez, and Francois Chollet. 2017 · 2017
Cited alongside, same era.
Six challenges for neural machine translation
Philipp Koehn and Rebecca Knowles. 2017 · 2017
Cited alongside, same era.
Analyzing neural MT search and model performance
Jan Niehues, Eunah Cho, Thanh-Le Ha, and Alex Waibel. 2017 · 2017
Cited alongside, same era.
The University of Cambridge’s machine translation systems for WMT18
Felix Stahlberg, Adrià de Gispert, and Bill Byrne. 2018a
Cited in the paper.
Why not be versatile? Applications of the SGNMT decoder for machine translation
Felix Stahlberg, Danielle Saunders, Gonzalo Iglesias, and Bill Byrne. 2018b
Cited in the paper.

SGNMT – A flexible NMT decoding platform for quick prototyping of new models and search strategies
Felix Stahlberg, Eva Hasler, Danielle Saunders, and Bill Byrne. 2017 · 2017
Later among the works it cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Later among the works it cites.
Recurrent neural networks as weighted language recognizers
Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. 2018 · 2018
Later among the works it cites.
Correcting length bias in neural machine translation
Kenton Murray and David Chiang. 2018 · 2018
Later among the works it cites.
Analyzing uncertainty in neural machine translation
Myle Ott, Michael Auli, David Grangier, et al. 2018 · 2018
Later among the works it cites.
Tensor2tensor for neural machine translation
Original
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018 · 2018
Later among the works it cites.
Breaking the beam search curse: A study of (re-) scoring methods and stopping criteria for neural machine translation
Yilin Yang, Liang Huang, and Mingbo Ma. 2018 · 2018
Later among the works it cites.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…