2020

Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation

Sun, Haipeng, Wang, Rui, Chen, Kehai et al.

Understand

Unsupervised neural machine translation (UNMT) has recently achieved remarkable results for several language pairs.

However, it can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time.
That is, research on multilingual UNMT has been limited.
In this paper, we empirically introduce a simple method to translate between thirteen languages using a single encoder and a single decoder, making use of multilingual data to improve UNMT for all language pairs.

Built on

Cross-lingual language model pretraining
Original
Guillaume Lample and Alexis Conneau. 2019 · 1901
Earlier work this paper cites.
Self-knowledge distillation in natural language processing
Original
Sangchul Hahn and Heeyoul Choi. 2019 · 1908
Earlier work this paper cites.
Moses: Open source toolkit for statistical machine translation
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007 · 2007
Earlier work this paper cites.
Ethnologue: Languages of the world
M Paul Lewis. 2009 · 2009
Earlier work this paper cites.
Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010 · 2010
Earlier work this paper cites.
Multi-task learning for multiple language translation
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015 · 2015
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015 · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. 2015 · 2015
Earlier work this paper cites.
Multi-way, multilingual neural machine translation with a shared attention mechanism
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016 · 2016
Earlier work this paper cites.
Toward multilingual neural machine translation with universal encoder and decoder
Original
Thanh-Le Ha, Jan Niehues, and Alexander H. Waibel. 2016 · 2016
Earlier work this paper cites.
Dual learning for machine translation
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016 · 2016
Earlier work this paper cites.

Similar

Learning distributed representations of sentences from unlabelled data
Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016 · 2016
Cited alongside, same era.
Multi-task sequence to sequence learning
Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016 · 2016
Cited alongside, same era.
Google’s multilingual neural machine translation system: Enabling zero-shot translation
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017 · 2017
Cited alongside, same era.
Unsupervised neural machine translation
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018 · 2018
Cited alongside, same era.
Multilingual neural machine translation with task-specific attention
Graeme Blackwood, Miguel Ballesteros, and Todd Ward. 2018 · 2018
Cited alongside, same era.
A comparison of transformer and recurrent neural networks on multilingual neural machine translation
Surafel Melaku Lakew, Mauro Cettolo, and Marcello Federico. 2018 · 2018
Cited alongside, same era.
A neural interlingua for multilingual machine translation
Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, and Jason Sun. 2018 · 2018
Cited alongside, same era.
Contextual parameter generation for universal neural machine translation
Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. 2018 · 2018
Cited alongside, same era.
Triangular architecture for rare language translation
Shuo Ren, Wenhu Chen, Shujie Liu, Mu Li, Ming Zhou, and Shuai Ma. 2018 · 2018
Cited alongside, same era.
Parameter sharing methods for multilingual self-attentional translation models
Devendra Sachan and Graham Neubig. 2018 · 2018
Cited alongside, same era.
Unsupervised neural machine translation with weight sharing
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2018 · 2018
Cited alongside, same era.
Unsupervised machine translation using monolingual corpora only
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a
Cited in the paper.

Then

Massively multilingual neural machine translation
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019 · 2019
Later among the works it cites.
Findings of the 2019 conference on machine translation (WMT19)
Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019 · 2019
Later among the works it cites.
The NiuTrans machine translation systems for WMT19
Bei Li, Yinqiao Li, Chen Xu, Ye Lin, Jiqiang Liu, Hui Liu, Ziyang Wang, Yuhao Zhang, Nuo Xu, Zeyang Wang, Kai Feng, Hexuan Chen, Tengbo Liu, Yanyang Li, Qiang Wang, Tong Xiao, and Jingbo Zhu. 2019 · 2019
Later among the works it cites.
NICT’s unsupervised neural and statistical machine translation systems for the WMT19 news translation task
Benjamin Marie, Haipeng Sun, Rui Wang, Kehai Chen, Atsushi Fujita, Masao Utiyama, and Eiichiro Sumita. 2019 · 2019
Later among the works it cites.
Multilingual unsupervised NMT using shared encoder and language-specific decoders
Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya. 2019 · 2019
Later among the works it cites.
Unsupervised bilingual word embedding agreement for unsupervised neural machine translation
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2019 · 2019
Later among the works it cites.
Target conditioned sampling: Optimizing data selection for multilingual neural machine translation
Xinyi Wang and Graham Neubig. 2019 · 2019
Later among the works it cites.
Polygon-net: A general framework for jointly boosting multiple unsupervised neural machine translation models
Chang Xu, Tao Qin, Gang Wang, and Tie-Yan Liu. 2019 · 2019
Later among the works it cites.
Unsupervised neural machine translation with cross-lingual language representation agreement
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2020 · 2020
Closest in time.
A survey of advances and challenges in unsupervised neural machine translation
Rui Wang, Haipeng Sun, and Sumita Eiichiro Utiyama, Masao. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…