2019

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges

Arivazhagan, Naveen, Bapna, Ankur, Firat, Orhan et al.

Understand

We introduce our efforts towards building a universal neural machine translation (NMT) system capable of translating between any language pair.

We set a milestone towards this goal by building a single massively multilingual NMT model handling 103 languages trained on over 25 billion examples.
Our system demonstrates effective transfer learning ability, significantly improving translation quality of low-resource languages, while keeping high-resource language translation quality on-par with competitive bilingual baselines.
We provide in-depth analysis of various aspects of model building that are crucial to achieving quality and practicality in universal NMT.

Built on

Pay less attention with lightweight and dynamic convolutions
Original
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. 2019 · 1901
Earlier work this paper cites.
Fixup initialization: Residual learning without normalization
Original
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. 2019 · 1901
Earlier work this paper cites.
Parameter-efficient transfer learning for nlp
Original
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019 · 1902
Earlier work this paper cites.
Lingvo: a modular and scalable framework for sequence-to-sequence modeling
Original
Jonathan Shen, Patrick Nguyen, Yonghui Wu, et al. 2019 · 1902
Earlier work this paper cites.
Multilingual neural machine translation with knowledge distillation
Original
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019 · 1902
Earlier work this paper cites.
Massively multilingual neural machine translation
Original
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019 · 1903
Earlier work this paper cites.
The missing ingredient in zero-shot neural machine translation
Original
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019 · 1903
Earlier work this paper cites.
Continual learning via neural pruning
Original
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. 2019 · 1903
Earlier work this paper cites.
Reinforcement learning based curriculum optimization for neural machine translation
Original
Gaurav Kumar, George Foster, Colin Cherry, and Maxim Krikun. 2019 · 1903
Earlier work this paper cites.
Competence-based curriculum learning for neural machine translation
Original
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom M. Mitchell. 2019 · 1903
Earlier work this paper cites.
Consistency by agreement in zero-shot neural machine translation
Original
Maruan Al-Shedivat and Ankur P Parikh. 2019 · 1904
Earlier work this paper cites.
Text repair model for neural machine translation
Original
Markus Freitag, Isaac Caswell, and Scott Roy. 2019 · 1904
Earlier work this paper cites.
Towards interlingua neural machine translation
Original
Carlos Escolano, Marta R Costa-jussà, and José AR Fonollosa. 2019 · 1905
Earlier work this paper cites.
Mass: Masked sequence to sequence pre-training for language generation
Original
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019 · 1905
Earlier work this paper cites.
Improved zero-shot neural machine translation via ignoring spurious correlations
Original
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2019 · 1906
Earlier work this paper cites.
Evaluating the supervised and zero-shot performance of multi-lingual translation models
Original
Chris Hokamp, John Glover, and Demian Gholipour. 2019 · 1906
Earlier work this paper cites.
Modeling task relationships in multi-task learning with multi-gate mixture-of-experts
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. 2018a · 1939
Earlier work this paper cites.
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014 · 1958
Earlier work this paper cites.
Art 2: Self-organization of stable category recognition codes for analog input patterns
Gail A Carpenter and Stephen Grossberg. 1987 · 1987
Earlier work this paper cites.
Neurocomputing: Foundations of research
Stephen Grossberg. 1988 · 1988
Earlier work this paper cites.
Lifelong robot learning
Sebastian Thrun and Tom M. Mitchell. 1995 · 1995
Earlier work this paper cites.
Multitask learning
Rich Caruana. 1997 · 1997
Earlier work this paper cites.
Machine learning
Tom M. Mitchell. 1997 · 1997
Earlier work this paper cites.
Learning to Learn
Sebastian Thrun and Lorien Pratt, editors. 1998 · 1998
Earlier work this paper cites.
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics
George Doddington. 2002 · 2002
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
A perspective view and survey of meta-learning
Ricardo Vilalta and Youssef Drissi. 2002 · 2002
Earlier work this paper cites.
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments
Satanjeev Banerjee and Alon Lavie. 2005 · 2005
Earlier work this paper cites.
Europarl: A Parallel Corpus for Statistical Machine Translation
Philipp Koehn. 2005 · 2005
Earlier work this paper cites.
To transfer or not to transfer
Michael T Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G Dietterich. 2005 · 2005
Earlier work this paper cites.
A unified architecture for natural language processing: Deep neural networks with multitask learning
Ronan Collobert and Jason Weston. 2008 · 2008
Earlier work this paper cites.
Curriculum learning
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009 · 2009
Earlier work this paper cites.
Tags for Identifying Languages
A. Phillips and M Davis. 2009 · 2009
Earlier work this paper cites.
Multi-domain learning by confidence-weighted parameter combination
Mark Dredze, Alex Kulesza, and Koby Crammer. 2010 · 2010
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio. 2010 · 2010
Earlier work this paper cites.
A survey on transfer learning
S. J. Pan and Q. Yang. 2010 · 2010
Earlier work this paper cites.
Large scale parallel document mining for machine translation
Jakob Uszkoreit, Jay M Ponte, Ashok C Popat, and Moshe Dubiner. 2010 · 2010
Earlier work this paper cites.
Learning with whom to share in multi-task feature learning
Zhuoliang Kang, Kristen Grauman, and Fei Sha. 2011 · 2011
Earlier work this paper cites.
One shot learning of simple visual concepts
Brenden M. Lake, Ruslan R. Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. 2011 · 2011
Earlier work this paper cites.
Wit3: Web inventory of transcribed and translated talks
Mauro Cettolo, C Girardi, and M Federico. 2012 · 2012
Earlier work this paper cites.
Multi-domain learning: When do domains matter?
Mahesh Joshi, Mark Dredze, William W. Cohen, and Carolyn Rose. 2012 · 2012
Earlier work this paper cites.
Parallel data, tools and interfaces in OPUS
Jörg Tiedemann. 2012 · 2012
Earlier work this paper cites.
New types of deep neural network learning for speech recognition and related applications: an overview
L. Deng, G. Hinton, and B. Kingsbury. 2013 · 2013
Earlier work this paper cites.
Recurrent continuous translation models
Nal Kalchbrenner and Phil Blunsom. 2013 · 2013
Earlier work this paper cites.
Lifelong machine learning systems: Beyond learning algorithms
Daniel L. Silver, Qiang Yang, and Lianghao Li. 2013 · 2013
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Learning phrase representations using rnn encoder–decoder for statistical machine translation
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Training deep neural networks with low precision multiplications
Original
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2014 · 2014
Earlier work this paper cites.
BEER: BEtter evaluation as ranking
Milos Stanojevic and Khalil Sima’an. 2014 · 2014
Earlier work this paper cites.

Similar

Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014 · 2014
Cited alongside, same era.
Semi-supervised sequence learning
Andrew M Dai and Quoc V Le. 2015 · 2015
Cited alongside, same era.
Multi-task learning for multiple language translation
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015 · 2015
Cited alongside, same era.
Multilingual language processing from bytes
Original
Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2015 · 2015
Cited alongside, same era.
On using monolingual corpora in neural machine translation
Original
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015 · 2015
Cited alongside, same era.
Character-based neural machine translation
Original
Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. 2015 · 2015
Cited alongside, same era.
Multi-task sequence to sequence learning
Original
Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015 · 2015
Cited alongside, same era.
Learning multi-domain convolutional neural networks for visual tracking
Original
Hyeonseob Nam and Bohyung Han. 2015 · 2015
Cited alongside, same era.
Curriculum learning of multiple tasks
Anastasia Pentina, Viktoriia Sharmanska, and Christoph H. Lampert. 2015 · 2015
Cited alongside, same era.
chrf: character n-gram f-score for automatic mt evaluation
Maja Popović. 2015 · 2015
Cited alongside, same era.
Massively multitask networks for drug discovery
Original
Bharath Ramsundar, Steven Kearnes, Patrick Riley, Dale Webster, David Konerding, and Vijay Pande. 2015 · 2015
Cited alongside, same era.
An embarrassingly simple approach to zero-shot learning
Bernardino Romera-Paredes and Philip H. S. Torr. 2015 · 2015
Cited alongside, same era.

Then

Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, et al. 2017 · 2017
Later among the works it cites.
Unsupervised machine translation using monolingual corpora only
Original
Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017 · 2017
Later among the works it cites.
Fully character-level neural machine translation without explicit segmentation
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017 · 2017
Later among the works it cites.
Gradient episodic memory for continual learning
David Lopez-Paz et al. 2017 · 2017
Later among the works it cites.
Transfer learning across low-resource, related languages for neural machine translation
Toan Q. Nguyen and David Chiang. 2017 · 2017
Later among the works it cites.
On the expressive power of deep neural networks
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl Dickstein. 2017 · 2017
Later among the works it cites.
An overview of multi-task learning in deep neural networks
Original
Sebastian Ruder. 2017 · 2017
Later among the works it cites.
Learning what to share between loosely related tasks
Original
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2017 · 2017
Later among the works it cites.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Original
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017 · 2017
Later among the works it cites.
Cold fusion: Training seq2seq models together with language models
Original
Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. 2017 · 2017
Later among the works it cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Later among the works it cites.
Dynamic data selection for neural machine translation
Original
Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017 · 2017
Later among the works it cites.
A survey on multi-task learning
Original
Yu Zhang and Qiang Yang. 2017 · 2017
Later among the works it cites.
On the optimization of deep networks: Implicit acceleration by overparameterization
Original
Sanjeev Arora, Nadav Cohen, and Elad Hazan. 2018 · 2018
Later among the works it cites.
Training deeper neural machine translation models with transparent attention
Original
Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018 · 2018
Later among the works it cites.
Multilingual neural machine translation with task-specific attention
Original
Graeme Blackwood, Miguel Ballesteros, and Todd Ward. 2018 · 2018
Later among the works it cites.
Findings of the 2018 conference on machine translation (wmt18)
Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018b · 2018
Later among the works it cites.
Large scale gan training for high fidelity natural image synthesis
Original
Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018 · 2018
Later among the works it cites.
Revisiting character-based neural machine translation with capacity and compression
Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. 2018 · 2018
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Later among the works it cites.
Understanding back-translation at scale
Original
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018 · 2018
Later among the works it cites.
Universal neural machine translation for extremely low resource languages
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K. Li. 2018a · 2018
Later among the works it cites.
Shampoo: Preconditioned stochastic tensor optimization
Original
Vineet Gupta, Tomer Koren, and Yoram Singer. 2018 · 2018
Later among the works it cites.
Achieving human parity on automatic chinese to english news translation
Original
Hany Hassan, Anthony Aue, Chang Chen, et al. 2018 · 2018
Later among the works it cites.
Gpipe: Efficient training of giant neural networks using pipeline parallelism
Original
Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, and Zhifeng Chen. 2018 · 2018
Later among the works it cites.
Adaptive scheduling for multi-task learning
Sébastien Jean, Orhan Firat, and Melvin Johnson. 2018 · 2018
Later among the works it cites.
Scheduled multi-task learning: From syntax to translation
Eliyahu Kiperwasser and Miguel Ballesteros. 2018 · 2018
Later among the works it cites.
Learning to segment inputs for NMT favors character-level processing
Original
Julia Kreutzer and Artem Sokolov. 2018 · 2018
Later among the works it cites.
SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson. 2018 · 2018
Later among the works it cites.
Transfer learning in multilingual neural machine translation with dynamic vocabulary
Original
Surafel Melaku Lakew, Aliia Erofeeva, Matteo Negri, Marcello Federico, and Marco Turchi. 2018 · 2018
Later among the works it cites.
A neural interlingua for multilingual machine translation
Original
Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, and Jason Sun. 2018 · 2018
Later among the works it cites.
Methods for interpreting and understanding deep neural networks
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. 2018 · 2018
Later among the works it cites.
Rapid adaptation of neural machine translation to new languages
Graham Neubig and Junjie Hu. 2018 · 2018
Later among the works it cites.
On first-order meta-learning algorithms
Original
Alex Nichol, Joshua Achiam, and John Schulman. 2018 · 2018
Later among the works it cites.
Scaling neural machine translation
Original
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018 · 2018
Later among the works it cites.
Continual lifelong learning with neural networks: A review
Original
German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kann, and Stefan Wermter. 2018 · 2018
Later among the works it cites.
Contextual parameter generation for universal neural machine translation
Original
Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. 2018 · 2018
Later among the works it cites.
A call for clarity in reporting bleu scores
Original
Matt Post. 2018 · 2018
Later among the works it cites.
When and why are pre-trained word embeddings useful for neural machine translation
Ye Qi, Sachan Devendra, Felix Matthieu, Padmanabhan Sarguna, and Neubig Graham. 2018 · 2018
Later among the works it cites.
Theory and experiments on vector quantized autoencoders
Original
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. 2018 · 2018
Later among the works it cites.
Parameter sharing methods for multilingual self-attentional translation models
Devendra Singh Sachan and Graham Neubig. 2018 · 2018
Later among the works it cites.
Mesh-tensorflow: Deep learning for supercomputers
Noam Shazeer, Youlong Cheng, Niki Parmar, et al. 2018 · 2018
Later among the works it cites.
Adafactor: Adaptive learning rates with sublinear memory cost
Original
Noam Shazeer and Mitchell Stern. 2018 · 2018
Later among the works it cites.
On training recurrent neural networks for lifelong learning
Original
Shagun Sodhani, Sarath Chandar, and Yoshua Bengio. 2018 · 2018
Later among the works it cites.
Emerging language spaces learned from massively multilingual corpora
Original
Jörg Tiedemann. 2018 · 2018
Later among the works it cites.
Three strategies to improve one-to-many multilingual translation
Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. 2018b · 2018
Later among the works it cites.
Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting
Original
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. 2019 · 2019
Closest in time.
Energy and policy considerations for deep learning in nlp
Original
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019 · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…