2020

Multilingual Translation with Extensible Multilingual Pretraining and Finetuning

Tang, Yuqing, Tran, Chau, Li, Xian et al.

Understand

Recent work demonstrates the potential of multilingual pretraining of creating one model that can be used for various tasks in different languages.

Previous work in multilingual pretraining has demonstrated that machine translation systems can be created by finetuning on bitext.
In this work, we show that multilingual translation models can be created through multilingual finetuning.
Instead of finetuning on one direction, a pretrained model is finetuned on many directions at the same time.

Built on

Cross-lingual language model pretraining
Original
Guillaume Lample and Alexis Conneau. 2019 · 1901
Earlier work this paper cites.
The missing ingredient in zero-shot neural machine translation
Original
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019a · 1903
Earlier work this paper cites.
Improved zero-shot neural machine translation via ignoring spurious correlations
Original
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2019 · 1906
Earlier work this paper cites.
Massively multilingual neural machine translation in the wild: Findings and challenges
Original
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019b · 1907
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019a · 1910
Earlier work this paper cites.
Mlqa: Evaluating cross-lingual extractive question answering
Original
Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019b · 1910
Earlier work this paper cites.
Unsupervised cross-lingual representation learning at scale
Original
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1911
Earlier work this paper cites.
Multilingual denoising pre-training for neural machine translation
Original
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020 · 2001
Earlier work this paper cites.

Similar

Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization
Original
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020 · 2003
Cited alongside, same era.
Improving massively multilingual neural machine translation and zero-shot translation
Original
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020 · 2004
Cited alongside, same era.
Leveraging monolingual data with self-supervision for multilingual neural machine translation
Original
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu. 2020 · 2005
Cited alongside, same era.
Cross-lingual retrieval for iterative self-supervised training
Original
Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu. 2020 · 2006
Cited alongside, same era.
Multi-way, multilingual neural machine translation with a shared attention mechanism
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016 · 2016
Cited alongside, same era.
Google’s multilingual neural machine translation system: Enabling zero-shot translation
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017 · 2017
Cited alongside, same era.
Bag of tricks for efficient text classification
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.

Then

XNLI: Evaluating cross-lingual sentence representations
Original
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 · 2018
Later among the works it cites.
Universal neural machine translation for extremely low resource languages
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K. Li. 2018 · 2018
Later among the works it cites.
SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson. 2018 · 2018
Later among the works it cites.
Deep contextualized word representations
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 · 2018
Later among the works it cites.
Improving language understanding with unsupervised learning
Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018 · 2018
Later among the works it cites.
Massively multilingual neural machine translation
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019 · 2019
Later among the works it cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Later among the works it cites.
MASS: Masked sequence to sequence pre-training for language generation
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019 · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…