2018

Correcting Length Bias in Neural Machine Translation

Murray, Kenton, Chiang, David

Understand

We study two problems in neural machine translation (NMT).

First, in beam search, whereas a wider beam should in principle help translation, it often hurts NMT.
Second, NMT has a tendency to produce translations that are too short.
Here, we argue that these problems are closely related and both rooted in label bias.

Built on

Une Approche théorique de l’Apprentissage Connexioniste; Applications à la reconnaissance de la Parole
Léon Bottou. 1991 · 1991
Earlier work this paper cites.
Conditional random fields: Probabilistic models for segmenting and labeling sequence data
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001 · 2001
Earlier work this paper cites.
Discriminative training and maximum entropy models for statistical machine translation
Franz Josef Och and Hermann Ney. 2002 · 2002
Earlier work this paper cites.
BLEU: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Minimum error rate training in statistical machine translation
Franz Josef Och. 2003 · 2003
Earlier work this paper cites.
Overview of the IWSLT 2010 evaluation campaign
Michael Paul, Marcello Federico, and Sebastian Stüker. 2010 · 2010
Earlier work this paper cites.
Analyzing the effect of global learning and beam-search on transition-based dependency parsing
Yue Zhang and Joakim Nivre. 2012 · 2012
Earlier work this paper cites.
Audio chord recognition with recurrent neural networks
Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. 2013 · 2013
Earlier work this paper cites.
Meteor universal: Language specific translation evaluation for any target language
Michael Denkowski and Alon Lavie. 2014 · 2014
Earlier work this paper cites.

Similar

Neural machine translation by jointly learning to align and translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015 · 2015
Cited alongside, same era.
Montreal neural machine translation systems for wmt’15
Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015 · 2015
Cited alongside, same era.
lamtram: A toolkit for language and translation modeling using neural networks
Graham Neubig. 2015 · 2015
Cited alongside, same era.
Sequence level training with recurrent neural networks
Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015 · 2015
Cited alongside, same era.
Improving multi-step prediction of learned time series models
Arun Venkatraman, Martial Hebert, and J Andrew Bagnell. 2015 · 2015
Cited alongside, same era.
Globally normalized transition-based neural networks
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016 · 2016
Cited alongside, same era.
Findings of the 2016 Conference on Machine Translation
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016 · 2016
Cited alongside, same era.
Improved neural machine translation with SMT features
Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016 · 2016
Cited alongside, same era.
Lexicons and minimum risk training for neural machine translation: NAIST-CMU at WAT2016
Graham Neubig. 2016 · 2016
Cited alongside, same era.

Then

Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Later among the works it cites.
Minimum risk training for neural machine translation
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016 · 2016
Later among the works it cites.
Sequence-to-sequence learning as beam-search optimization
Sam Wiseman and Alexander M Rush. 2016 · 2016
Later among the works it cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016 · 2016
Later among the works it cites.
Findings of the 2017 Conference on Machine Translation (WMT17)
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. 2017 · 2017
Later among the works it cites.
Stronger baselines for trustable results in neural machine translation
Michael Denkowski and Graham Neubig. 2017 · 2017
Later among the works it cites.
When to finish? optimal beam search for neural text generation (modulo beam size)
Liang Huang, Kai Zhao, and Mingbo Ma. 2017 · 2017
Later among the works it cites.
Six challenges for neural machine translation
Philipp Koehn and Rebecca Knowles. 2017 · 2017
Later among the works it cites.
Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation
Yilin Yang, Liang Huang, and Mingbo Ma. 2018 · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…