2020

Cross-lingual Retrieval for Iterative Self-Supervised Training

Tran, Chau, Tang, Yuqing, Li, Xian et al.

Understand

Recent studies have demonstrated the cross-lingual alignment ability of multilingual pretrained language models.

In this work, we found that the cross-lingual alignment can be further improved by training seq2seq models on sentence pairs mined using their own encoder outputs.
We utilized these findings to develop a new approach -- cross-lingual retrieval for iterative self-supervised training (CRISS), where mining and training processes are applied iteratively, improving cross-lingual alignment and translation ability at the same time.
Using this method, we achieved state-of-the-art unsupervised machine translation results on 9 language directions with an average improvement of 2.4 BLEU, and on the Tatoeba sentence retrieval task in the XTREME benchmark on 16 languages with an average improvement of 21.5% in absolute accuracy.

Built on

Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002
Earlier work this paper cites.
Large scale parallel document mining for machine translation
Jakob Uszkoreit, Jay Ponte, Ashok Popat, and Moshe Dubiner · 2010
Earlier work this paper cites.
Findings of the 2013 Workshop on Statistical Machine Translation
Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia · 2013
Earlier work this paper cites.
Exploiting similarities among languages for machine translation
Original
Tomas Mikolov, Quoc V Le, and Ilya Sutskever · 2013
Earlier work this paper cites.
The iwslt 2015 evaluation campaign
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, R. Cattoni, and Marcello Federico · 2015
Earlier work this paper cites.
Learning principled bilingual mappings of word embeddings while preserving monolingual invariance
Mikel Artetxe, Gorka Labaka, and Eneko Agirre · 2016
Earlier work this paper cites.
Findings of the 2016 conference on machine translation
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al · 2016
Earlier work this paper cites.
Improving neural machine translation models with monolingual data
Rico Sennrich, Barry Haddow, and Alexandra Birch · 2016
Earlier work this paper cites.
Overview of the IWSLT 2017 evaluation campaign
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker Sebastian, Sudoh Katsuitho, Yoshino Koichiro, and Federmann Christian · 2017
Earlier work this paper cites.
Unsupervised machine translation using monolingual corpora only
Original
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato · 2017
Earlier work this paper cites.
Findings of the 2017 conference on machine translation (wmt17)
Bojar Ondrej, Rajen Chatterjee, Federmann Christian, Graham Yvette, Haddow Barry, Huck Matthias, Koehn Philipp, Liu Qun, Logacheva Varvara, Monz Christof, et al · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora
Hainan Xu and Philipp Koehn · 2017
Earlier work this paper cites.
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
Mikel Artetxe, Gorka Labaka, and Eneko Agirre · 2018
Earlier work this paper cites.
Findings of the 2018 conference on machine translation (WMT18)
Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz · 2018
Earlier work this paper cites.
Effective parallel corpus mining using bilingual sentence embeddings
Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, et al · 2018
Earlier work this paper cites.
The JHU parallel corpus filtering systems for WMT 2018
Huda Khayrallah, Hainan Xu, and Philipp Koehn · 2018
Earlier work this paper cites.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson · 2018
Earlier work this paper cites.
The iit bombay english-hindi parallel corpus
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya · 2018
Earlier work this paper cites.
Word translation without parallel data
Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou · 2018
Earlier work this paper cites.

Similar

Deep contextualized word representations
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer · 2018
Cited alongside, same era.
When and why are pre-trained word embeddings useful for neural machine translation?
Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig · 2018
Cited alongside, same era.
Improving language understanding with unsupervised learning
Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever · 2018
Cited alongside, same era.
Filtering and mining parallel data in a joint multilingual space
Holger Schwenk · 2018
Cited alongside, same era.
Unsupervised cross-lingual word embedding by multilingual neural language models
Original
Takashi Wada and Tomoharu Iwata · 2018
Cited alongside, same era.
Context-aware cross-lingual mapping
Hanan Aldarmaki and Mona Diab · 2019
Cited alongside, same era.
Bilingual lexicon induction through unsupervised machine translation
Mikel Artetxe, Gorka Labaka, and Eneko Agirre · 2019
Cited alongside, same era.
Margin-based parallel corpus mining with multilingual sentence embeddings
Mikel Artetxe and Holger Schwenk · 2019
Cited alongside, same era.
Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond
Mikel Artetxe and Holger Schwenk · 2019
Cited alongside, same era.
Findings of the 2019 conference on machine translation (wmt19)
Loïc Barrault, Ondřej Bojar, Marta R Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, et al · 2019
Cited alongside, same era.
Low-resource corpus filtering using multilingual sentence embeddings
Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn · 2019
Cited alongside, same era.
Unsupervised cross-lingual representation learning at scale
Original
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Cited alongside, same era.

Then

Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer · 2019
Later among the works it cites.
Data-dependent gaussian prior objective for language generation
Zuchao Li, Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao · 2019
Later among the works it cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Later among the works it cites.
Overview of the 6th workshop on Asian translation
Toshiaki Nakazawa, Nobushige Doi, Shohei Higashiyama, Chenchen Ding, Raj Dabre, Hideya Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan, Shantipriya Parida, Ondřej Bojar, and Sadao Kurohashi · 2019
Later among the works it cites.
fairseq: A fast, extensible toolkit for sequence modeling
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli · 2019
Later among the works it cites.
How multilingual is multilingual bert?
Telmo Pires, Eva Schlinger, and Dan Garrette · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Original
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu · 2019
Later among the works it cites.
Explicit cross-lingual pre-training for unsupervised machine translation
Shuo Ren, Yu Wu, Shujie Liu, Ming Zhou, and Shuai Ma · 2019
Later among the works it cites.
Ccmatrix: Mining billions of high-quality parallel sentences on the web
Original
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin · 2019
Later among the works it cites.
Mass: Masked sequence to sequence pre-training for language generation
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu · 2019
Later among the works it cites.
Cross-lingual ability of multilingual bertg: An empirical study
Original
Zihan Wang, Stephen Mayhew, Dan Roth, et al · 2019
Later among the works it cites.
Ccnet: Extracting high quality monolingual datasets from web crawl data
Original
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave · 2019
Later among the works it cites.
Extract and edit: An alternative to back-translation for unsupervised neural machine translation
Jiawei Wu, Xin Wang, and William Yang Wang · 2019
Later among the works it cites.
Emerging cross-lingual structure in pretrained language models
Original
Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Later among the works it cites.
XLNET: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le · 2019
Later among the works it cites.
A multilingual view of unsupervised machine translation
Original
Xavier Garcia, Pierre Foret, Thibault Sellam, and Ankur P Parikh · 2020
Closest in time.
Realm: Retrieval-augmented language model pre-training
Original
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang · 2020
Closest in time.
Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization
Original
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson · 2020
Closest in time.
Multilingual denoising pre-training for neural machine translation
Original
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…