2017

Dynamic Data Selection for Neural Machine Translation

van der Wees, Marlies, Bisazza, Arianna, Monz, Christof

Understand

Intelligent selection of training data has proven a successful technique to simultaneously increase training efficiency and translation performance for phrase-based machine translation (PBMT).

With the recent increase in popularity of neural machine translation (NMT), we explore in this paper to what extent and how NMT can also benefit from data selection.
While state-of-the-art data selection (Axelrod et al., 2011) consistently performs well for PBMT, we show that gains are substantially lower for NMT.
Next, we introduce dynamic data selection for NMT, a method in which we vary the selected subset of training data between different training epochs.

Built on

Toward a unified approach to statistical language modeling for chinese
Jianfeng Gao, Joshua Goodman, Mingjing Li, and Kai-Fu Lee. 2002 · 2002
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Low cost portability for statistical machine translation based on n-gram frequency and TF-IDF
Matthias Eck, Stephan Vogel, and Alex Waibel. 2005 · 2005
Earlier work this paper cites.
Moses: Open source toolkit for statistical machine translation
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007 · 2007
Earlier work this paper cites.
Method of selecting training data to build a compact and efficient translation model
Keiji Yasuda, Ruiqiang Zhang, Hirofumi Yamamoto, and Eiichiro Sumit. 2008 · 2008
Earlier work this paper cites.
News from OPUS-a collection of multilingual parallel corpora with tools and interfaces
Jörg Tiedemann. 2009 · 2009
Earlier work this paper cites.
Intelligent selection of language model training data
Robert C. Moore and William Lewis. 2010 · 2010
Earlier work this paper cites.
Domain adaptation via pseudo in-domain data selection
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011 · 2011
Earlier work this paper cites.
Tuning as ranking
Mark Hopkins and Jonathan May. 2011 · 2011
Earlier work this paper cites.
Wit 3 : Web inventory of transcribed and translated talks
Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012 · 2012
Earlier work this paper cites.
Does more data always yield better translations?
Guillem Gascó, Martha-Alicia Rocha, Germán Sanchis-Trilles, Jesús Andrés-Ferrer, and Francisco Casacuberta. 2012 · 2012
Earlier work this paper cites.
Adaptation data selection using neural language models: Experiments in machine translation
Kevin Duh, Graham Neubig, Katsuhito Sudoh, and Hajime Tsukada. 2013 · 2013
Earlier work this paper cites.
Dramatically reducing training data size through vocabulary saturation
William D. Lewis and Sauleh Eetemadi. 2013 · 2013
Earlier work this paper cites.

Similar

Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Cited alongside, same era.
Learning phrase representations using RNN encoder–decoder for statistical machine translation
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Cited alongside, same era.
Data selection for compact adapted SMT models
Shachar Mirkin and Laurent Besacier. 2014 · 2014
Cited alongside, same era.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014 · 2014
Cited alongside, same era.
Data selection with fewer words
Amittai Axelrod, Philip Resnik, Xiaodong He, and Mari Ostendorf. 2015 · 2015
Cited alongside, same era.
Survey of data-selection methods in statistical machine translation
Sauleh Eetemadi, William Lewis, Kristina Toutanova, and Hayder Radha. 2015 · 2015
Cited alongside, same era.
Stanford neural machine translation systems for spoken language domains
Minh-Thang Luong and Christopher D Manning. 2015 · 2015
Cited alongside, same era.
Effective approaches to attention-based neural machine translation
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a · 2015
Cited alongside, same era.
Neural versus phrase-based machine translation quality: a case study
Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016 · 2016
Cited alongside, same era.
Findings of the 2016 conference on machine translation (WMT16)
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016 · 2016
Cited alongside, same era.
Semi-supervised convolutional networks for translation adaptation with tiny amount of in-domain data
Boxing Chen and Fei Huang. 2016 · 2016
Cited alongside, same era.
SYSTRAN’s pure neural machine translation systems
Original
Josep Crego, Jungi Kim, Guillaume Klein, Anabel Rebollo, Kathy Yang, Jean Senellart, Egor Akhanov, Patrice Brunelle, Aurelien Coquard, Yongchao Deng, et al. 2016 · 2016
Cited alongside, same era.

Then

Neural machine translation in linear time
Original
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2016 · 2016
Later among the works it cites.
Sequence-level knowledge distillation
Yoon Kim and Alexander M. Rush. 2016 · 2016
Later among the works it cites.
Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles
Pierre Lison and Jörg Tiedemann. 2016 · 2016
Later among the works it cites.
Measuring the effect of conversational aspects on machine translation quality
Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2016 · 2016
Later among the works it cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016 · 2016
Later among the works it cites.
Boosting neural machine translation
Original
Dakun Zhang, Jungi Kim, Joseph Crego, and Jean Senellart. 2016 · 2016
Later among the works it cites.
Transfer learning for low-resource neural machine translation
Original
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016 · 2016
Later among the works it cites.
An empirical comparison of simple domain adaptation methods for neural machine translation
Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017 · 2017
Closest in time.
Data augmentation for low-resource neural machine translation
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017 · 2017
Closest in time.
Neural vs. phrase-based machine translation in a multi-domain scenario
M. Amin Farajian, Marco Turchi, Matteo Negri, Nicola Bertoldi, and Marcello Federico. 2017 · 2017
Closest in time.
Six challenges for neural machine translation
Original
Philipp Koehn and Rebecca Knowles. 2017 · 2017
Closest in time.
Learning to select data for transfer learning with bayesian optimization
Sebastian Ruder and Barbara Plank. 2017 · 2017
Closest in time.
A multifaceted evaluation of neural versus phrase-based machine translation for 9 language directions
Antonio Toral and Víctor M. Sánchez-Cartagena. 2017 · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…