2019

Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring

Liu, Zihan, Xu, Yan, Winata, Genta Indra et al.

Understand

This paper describes CAiRE's submission to the unsupervised machine translation track of the WMT'19 news shared task from German to Czech.

We leverage a phrase-based statistical machine translation (PBSMT) model and a pre-trained language model to combine word-level neural machine translation (NMT) and subword-level NMT models without using any parallel data.
We propose to solve the morphological richness problem of languages by training byte-pair encoding (BPE) embeddings for German and Czech separately, and they are aligned using MUSE (Conneau et al., 2018).
To ensure the fluency and consistency of translations, a rescoring mechanism is proposed that reuses the pre-trained language model to select the translation candidates generated through beam search.

Built on

Cross-lingual language model pretraining
Original
Guillaume Lample and Alexis Conneau. 2019 · 1901
Earlier work this paper cites.
Minimum error rate training in statistical machine translation
Franz Josef Och. 2003 · 2003
Earlier work this paper cites.
Moses: Open source toolkit for statistical machine translation
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007 · 2007
Earlier work this paper cites.
Statistical parsing of morphologically rich languages (spmrl): what, how and whither
Reut Tsarfaty, Djamé Seddah, Yoav Goldberg, Sandra Kübler, Marie Candito, Jennifer Foster, Yannick Versley, Ines Rehbein, and Lamia Tounsi. 2010 · 2010
Earlier work this paper cites.
Kenlm: Faster and smaller language model queries
Kenneth Heafield. 2011 · 2011
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015 · 2015
Earlier work this paper cites.
Unsupervised representation learning with deep convolutional generative adversarial networks
Alec Radford, Luke Metz, and Soumith Chintala. 2016 · 2016
Earlier work this paper cites.

Similar

Learning bilingual word embeddings with (almost) no bilingual data
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017 · 2017
Cited alongside, same era.
Enriching word vectors with subword information
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017 · 2017
Cited alongside, same era.
Convolutional sequence to sequence learning
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
Unsupervised statistical machine translation
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a · 2018
Cited alongside, same era.
Word translation without parallel data
Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018 · 2018
Cited alongside, same era.
Unsupervised neural machine translation
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018b
Cited in the paper.
Unsupervised machine translation using monolingual corpora only
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a
Cited in the paper.
An Analysis of Neural Language Modeling at Multiple Scales
Original
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018a
Cited in the paper.
Regularizing and optimizing LSTM language models
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018b
Cited in the paper.
Improving neural machine translation models with monolingual data
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a
Cited in the paper.
Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b
Cited in the paper.

Then

Are all languages equally hard to language-model?
Ryan Cotterell, Sebastian J Mielke, Jason Eisner, and Brian Roark. 2018 · 2018
Later among the works it cites.
Universal neural machine translation for extremely low resource languages
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li. 2018 · 2018
Later among the works it cites.
Phrase-based & neural unsupervised machine translation
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018b · 2018
Later among the works it cites.
Tensor2tensor for neural machine translation
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, et al. 2018 · 2018
Later among the works it cites.
Team yeon-zi at semeval-2019 task 4: Hyperpartisan news detection by de-noising weakly-labeled data
Nayeon Lee, Zihan Liu, and Pascale Fung. 2019 · 2019
Closest in time.
Extract and edit: An alternative to back-translation for unsupervised neural machine translation
Jiawei Wu, Xin Wang, and William Yang Wang. 2019 · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…