2020

Beyond English-Centric Multilingual Machine Translation

Fan, Angela, Bhosale, Shruti, Schwenk, Holger et al.

Understand

Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages.

However, much of this work is English-Centric by training only on data which was translated from or to English.
While this is supported by large sources of training data, it does not reflect translation needs worldwide.
In this work, we create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages.

Built on

Mining the Web for Bilingual Text
Philip Resnik · 1999
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002
Earlier work this paper cites.
Reliable Measures for Aligning Japanese-English News Articles and Sentences
Masao Utiyama and Hitoshi Isahara · 2003
Earlier work this paper cites.
Europarl: A parallel corpus for statistical machine translation
Philipp Koehn · 2005
Earlier work this paper cites.
Moses: Open source toolkit for statistical machine translation
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al · 2007
Earlier work this paper cites.
Investigations on large-scale lightly-supervised training for statistical machine translation
Holger Schwenk · 2008
Earlier work this paper cites.
The United Nations parallel corpus v1.0
Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen · 2010
Earlier work this paper cites.
Improving translation model by monolingual data
Ondrej Bojar and Ales Tamchyna · 2011
Earlier work this paper cites.
Parallel data, tools and interfaces in OPUS
Jörg Tiedemann · 2012
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2014
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
DP Kingma and LJ Ba · 2015
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Original
Rico Sennrich, Barry Haddow, and Alexandra Birch · 2015
Earlier work this paper cites.
Rethinking the Inception Architecture for Computer Vision
Original
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna · 2015
Earlier work this paper cites.
Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Earlier work this paper cites.
Training deep nets with sublinear memory cost
Original
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin · 2016
Earlier work this paper cites.
Similar southeast asian languages: Corpus-based case study on thai-laotian and malay-indonesian
Chenchen Ding, Masao Utiyama, and Eiichiro Sumita · 2016
Earlier work this paper cites.
Multi-way, multilingual neural machine translation with a shared attention mechanism
Original
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio · 2016
Earlier work this paper cites.
Ensemble learning for multi-source neural machine translation
Ekaterina Garmash and Christof Monz · 2016
Earlier work this paper cites.
Toward multilingual neural machine translation with universal encoder and decoder
Original
Thanh-Le Ha, Jan Niehues, and Alexander Waibel · 2016
Earlier work this paper cites.
PyThaiNLP: Thai Natural Language Processing in Python, June 2016
Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriyawongkul, Lalita Lowphansirikul, and Pattarawat Chormai · 2016
Earlier work this paper cites.
Introduction of the asian language treebank
Hammam Riza, Michael Purwoadi, Teduh Uliniansyah, Aw Ai Ti, Sharifah Mahani Aljunied, Luong Chi Mai, Vu Tat Thang, Nguyen Phuong Thai, Vichet Chea, Sethserey Sam, et al · 2016
Earlier work this paper cites.
Overview of the iwslt 2017 evaluation campaign
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker Sebastian, Sudoh Katsuitho, Yoshino Koichiro, and Federmann Christian · 2017
Earlier work this paper cites.
Convolutional Sequence to Sequence Learning
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin · 2017
Earlier work this paper cites.
Google’s multilingual neural machine translation system: Enabling zero-shot translation
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al · 2017
Earlier work this paper cites.
Findings of the 2017 conference on machine translation (wmt17)
Bojar Ondrej, Rajen Chatterjee, Federmann Christian, Graham Yvette, Haddow Barry, Huck Matthias, Koehn Philipp, Liu Qun, Logacheva Varvara, Monz Christof, et al · 2017
Earlier work this paper cites.
Regularizing neural networks by penalizing confident output distributions
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton · 2017
Earlier work this paper cites.
Tilde’s machine translation systems for WMT 2017
Mārcis Pinnis, Rihards Krišlauks, Toms Miks, Daiga Deksne, and Valters Šics · 2017
Earlier work this paper cites.
The University of Edinburgh’s neural MT systems for WMT17
Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.

Similar

Parallel corpora for bi-directional statistical machine translation for seven ethiopian language pairs
Solomon Teferra Abate, Michael Melese, Martha Yifiru Tachbelie, Million Meshesha, Solomon Atinafu, Wondwossen Mulugeta, Yaregal Assabie, Hafte Abera, Binyam Ephrem Seyoum, Tewodros Abebe, et al · 2018
Cited alongside, same era.
On the optimization of deep networks: Implicit acceleration by overparameterization
Original
Sanjeev Arora, Nadav Cohen, and Elad Hazan · 2018
Cited alongside, same era.
Findings of the 2018 conference on machine translation (WMT18)
Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz · 2018
Cited alongside, same era.
Understanding back-translation at scale
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier · 2018
Cited alongside, same era.
Universal neural machine translation for extremely low resource languages
Original
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li · 2018
Cited alongside, same era.
Iterative back-translation for neural machine translation
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn · 2018
Cited alongside, same era.
Findings of the wmt 2018 shared task on parallel corpus filtering
Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L Forcada · 2018
Cited alongside, same era.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Original
Taku Kudo and John Richardson · 2018
Cited alongside, same era.
Tilde’s machine translation systems for WMT 2018
Mārcis Pinnis, Matīss Rikters, and Rihards Krišlauks · 2018
Cited alongside, same era.
A call for clarity in reporting BLEU scores
Matt Post · 2018
Cited alongside, same era.
Mesh-tensorflow: Deep learning for supercomputers
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al · 2018
Cited alongside, same era.
Three strategies to improve one-to-many multilingual translation
Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong · 2018
Cited alongside, same era.

Then

Facebook FAIR’s WMT19 news translation task submission
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov · 2019
Later among the works it cites.
Transformers without tears: Improving the normalization of self-attention
Original
Toan Q Nguyen and Julian Salazar · 2019
Later among the works it cites.
Tilde’s machine translation systems for WMT 2019
Marcis Pinnis, Rihards Krišlauks, and Matīss Rikters · 2019
Later among the works it cites.
Zero: Memory optimizations toward training trillion parameter models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He · 2019
Later among the works it cites.
CCMatrix: Mining billions of high-quality parallel sentences on the web
Original
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan · 2019
Later among the works it cites.
Revisiting low-resource neural machine translation: A case study
Rico Sennrich and Biao Zhang · 2019
Later among the works it cites.
The source-target domain mismatch problem in machine translation
Original
Jiajun Shen, Peng-Jen Chen, Matt Le, Junxian He, Jiatao Gu, Myle Ott, Michael Auli, and Marc’Aurelio Ranzato · 2019
Later among the works it cites.
Megatron-lm: Training multi-billion parameter language models using gpu model parallelism
Original
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro · 2019
Later among the works it cites.
The University of Helsinki submissions to the WMT19 news translation task
Aarne Talman, Umut Sulubacak, Raúl Vázquez, Yves Scherrer, Sami Virpioja, Alessandro Raganato, Arvi Hurskainen, and Jörg Tiedemann · 2019
Later among the works it cites.
Ccnet: Extracting high quality monolingual datasets from web crawl data
Original
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave · 2019
Later among the works it cites.
Pay less attention with lightweight and dynamic convolutions
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli · 2019
Later among the works it cites.
Towards supervised and unsupervised neural machine translation baselines for nigerian pidgin
Original
Orevaoghene Ahia and Kelechi Ogueji · 2020
Closest in time.
Using lstm to translate french to senegalese local languages: Wolof as a case study
Original
Lo Alla, Dione Cheikh Bamba, Nguer Elhadji Mamadou, Ba Sileye O Ba, and Lo Moussa · 2020
Closest in time.
Syntactic relevance xlnet word embedding generation in low-resource machine translation
Anonymous · 2020
Closest in time.
Parallel sentence mining by constrained decoding
Pinzhen Chen, Nikolay Bogoychev, Kenneth Heafield, and Faheem Kirefu · 2020
Closest in time.
CCAligned: A massive collection of cross-lingual web-document pairs
Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzman, and Philipp Koehn · 2020
Closest in time.
Ffr v1. 1: Fon-french neural machine translation
Chris Chinenye Emezue and Femi Pancrace Bonaventure Dossou · 2020
Closest in time.
Igbo-english machine translation: An evaluation benchmark
Original
Ignatius Ezeani, Paul Rayson, Ikechukwu Onyenwe, Chinedu Uchechukwu, and Mark Hepple · 2020
Closest in time.
Participatory research for low-resourced machine translation: A case study in african languages
Original
∀ \forall , Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Tajudeen Kolawole, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddee Hassan Muhammad, Salomon Kabongo, Salomey Osei, et al · 2020
Closest in time.
Scaling laws for neural language models
Original
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2020
Closest in time.
Deep encoder, shallow decoder: Reevaluating the speed-quality tradeoff in machine translation
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith · 2020
Closest in time.
torchgpipe: On-the-fly pipeline parallelism for training giant models
Original
Chiheon Kim, Heungsub Lee, Myungryong Jeong, Woonhyuk Baek, Boogeon Yoon, Ildoo Kim, Sungbin Lim, and Sungwoong Kim · 2020
Closest in time.
The IndicNLP Library
Anoop Kunchukuttan · 2020
Closest in time.
Unsupervised multilingual sentence embeddings for parallel corpus mining
Ivana Kvapilíková, Mikel Artetxe, Gorka Labaka amd Eneko Agirre, and Ondřej Bojar · 2020
Closest in time.
Gshard: Scaling giant models with conditional computation and automatic sharding
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen · 2020
Closest in time.
Multilingual denoising pre-training for neural machine translation
Original
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer · 2020
Closest in time.
Leveraging monolingual data with self-supervision for multilingual neural machine translation
Original
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu · 2020
Closest in time.
Balancing training for multilingual neural machine translation
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig · 2020
Closest in time.
Improving massively multilingual neural machine translation and zero-shot translation
Original
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…