2022

Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?

Lee, En-Shiun Annie, Thillainathan, Sarubi, Nayak, Shravan et al.

Understand

What can pre-trained multilingual sequence-to-sequence models like mBART contribute to translating low-resource languages? We conduct a thorough empirical experiment in 10 languages to ascertain this, considering five factors: (1) the amount of fine-tuning data, (2) the noise in the fine-tuning data, (3) the amount of pre-training data in the model, (4) the impact of domain mismatch, and (5) language typology.

In addition to yielding several heuristics, the experiments form a framework for evaluating the data sensitivities of machine translation systems.
While mBART is robust to domain differences, its translations for unseen and typologically distant languages remain below 3.0 BLEU.
In answer to our title's question, mBART is not a low-resource panacea; we therefore encourage shifting the emphasis from new models to new data.

Built on

Massively multilingual neural machine translation in the wild: Findings and challenges
Original
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019 · 1907
Earlier work this paper cites.
PMIndia - A collection of parallel corpora of languages of India
Original
Barry Haddow and Faheem Kirefu. 2020 · 2001
Earlier work this paper cites.
Data augmentation and terminology integration for domain-specific Sinhala-English-Tamil statistical machine translation
Original
Aloka Fernando, Surangika Ranathunga, and Gihan Dias. 2020 · 2011
Earlier work this paper cites.
Parallel data, tools and interfaces in OPUS
Jörg Tiedemann. 2012 · 2012
Earlier work this paper cites.
WALS Online
Matthew S. Dryer and Martin Haspelmath, editors. 2013 · 2013
Earlier work this paper cites.
Beyond bitext: Five open problems in machine translation
Adam Lopez and Matt Post. 2013 · 2013
Earlier work this paper cites.
chrF: character n-gram F-score for automatic MT evaluation
Maja Popović. 2015 · 2015
Earlier work this paper cites.
An empirical study of language relatedness for transfer learning in neural machine translation
Raj Dabre, Tetsuji Nakagawa, and Hideto Kazawa. 2017 · 2017
Earlier work this paper cites.
Six challenges for neural machine translation
Philipp Koehn and Rebecca Knowles. 2017 · 2017
Earlier work this paper cites.
URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
On the impact of various types of noise on neural machine translation
Huda Khayrallah and Philipp Koehn. 2018 · 2018
Earlier work this paper cites.
Rapid adaptation of neural machine translation to new languages
Graham Neubig and Junjie Hu. 2018 · 2018
Earlier work this paper cites.
A call for clarity in reporting BLEU scores
Matt Post. 2018 · 2018
Earlier work this paper cites.
Filtering and mining parallel data in a joint multilingual space
Holger Schwenk. 2018 · 2018
Earlier work this paper cites.

Similar

Benchmarking neural machine translation for Southern African languages
Jade Abbott and Laura Martinus. 2019 · 2019
Cited alongside, same era.
JW300: A wide-coverage parallel corpus for low-resource languages
Željko Agić and Ivan Vulić. 2019 · 2019
Cited alongside, same era.
Evaluating the supervised and zero-shot performance of multi-lingual translation models
Chris Hokamp, John Glover, and Demian Gholipour Ghalandari. 2019 · 2019
Cited alongside, same era.
Towards realistic practices in low-resource natural language processing: The development set
Katharina Kann, Kyunghyun Cho, and Samuel R. Bowman. 2019 · 2019
Cited alongside, same era.
Investigating multilingual NMT representations at scale
Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019 · 2019
Cited alongside, same era.
How multilingual is multilingual BERT?
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019 · 2019
Cited alongside, same era.
Massive vs. curated embeddings for low-resourced languages: the case of Yorùbá and Twi
Jesujoba Alabi, Kwabena Amponsah-Kaakyire, David Adelani, and Cristina España-Bonet. 2020 · 2020
Cited alongside, same era.
CCAligned: A massive collection of cross-lingual web-document pairs
Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020 · 2020
Cited alongside, same era.
XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020 · 2020
Cited alongside, same era.
The state and fate of linguistic diversity and inclusion in the NLP world
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020 · 2020
Cited alongside, same era.
Multilingual denoising pre-training for neural machine translation
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020 · 2020
Cited alongside, same era.
Transfer learning for related languages: Submissions to the WMT20 similar language translation task
Lovish Madaan, Soumya Sharma, and Parag Singla. 2020 · 2020
Cited alongside, same era.

Then

The effect of domain and diacritics in Yoruba–English neural machine translation
David Adelani, Dana Ruiter, Jesujoba Alabi, Damilola Adebonojo, Adesina Ayeni, Mofe Adeyemi, Ayodele Esther Awokoya, and Cristina España-Bonet. 2021 · 2021
Later among the works it cites.
Indonlg: Benchmark and resources for evaluating indonesian natural language generation
Original
Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Leylia Khodra, et al. 2021 · 2021
Later among the works it cites.
How to adapt your pretrained multilingual model to 1600 languages
Abteen Ebrahimi and Katharina Kann. 2021 · 2021
Later among the works it cites.
Beyond English-centric multilingual machine translation
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. 2021 · 2021
Later among the works it cites.
Continual mixed-language pre-training for extremely low-resource neural machine translation
Zihan Liu, Genta Indra Winata, and Pascale Fung. 2021b · 2021
Later among the works it cites.
Findings of the AmericasNLP 2021 shared task on open machine translation for indigenous languages of the Americas
Manuel Mager, Arturo Oncevay, Abteen Ebrahimi, John Ortega, Annette Rios, Angela Fan, Ximena Gutierrez-Vasques, Luis Chiruzzo, Gustavo Giménez-Lugo, Ricardo Ramos, Ivan Vladimir Meza Ruiz, Rolando Coto-Solano, Alexis Palmer, Elisabeth Mager-Hois, Vishrav Chaudhary, Graham Neubig, Ngoc Thang Vu, and Katharina Kann. 2021 · 2021
Later among the works it cites.
Neural machine translation for low-resource languages: A survey
Original
Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2021 · 2021
Later among the works it cites.
CCMatrix: Mining billions of high-quality parallel sentences on the web
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021 · 2021
Later among the works it cites.
Multilingual translation from denoising pre-training
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021 · 2021
Later among the works it cites.
Fine-tuning self-supervised multilingual sequence-to-sequence models for extremely low-resource NMT
Sarubi Thillainathan, Surangika Ranathunga, and Sanath Jayasena. 2021 · 2021
Later among the works it cites.
Analyzing the source and target contributions to predictions in neural machine translation
Elena Voita, Rico Sennrich, and Ivan Titov. 2021 · 2021
Later among the works it cites.
mT5: A massively multilingual pre-trained text-to-text Transformer
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021 · 2021
Later among the works it cites.
Family of origin and family of choice: Massively parallel lexiconized iterative pretraining for severely low resource text-based translation
Zhong Zhou and Alexander Waibel. 2021 · 2021
Later among the works it cites.
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…