2023

When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages

Chang, Tyler A., Arnett, Catherine, Tu, Zhuowen et al.

Understand

Multilingual language models are widely used to extend NLP systems to low-resource languages.

However, concrete evidence for the effects of multilinguality on language modeling performance in individual languages remains scarce.
Here, we pre-train over 10,000 monolingual and multilingual language models for over 250 languages, including multiple language families that are under-studied in NLP.
We assess how language modeling performance in each language varies as a function of (1) monolingual dataset size, (2) added multilingual dataset size, (3) linguistic similarity of the added languages, and (4) model size (up to 45M parameters).

Built on

Well-read students learn better: On the importance of pre-training compact models
Original
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 1908
Earlier work this paper cites.
Teoria statistica delle classi ecalcolo delle probabilità
C.E. Bonferroni · 1936
Earlier work this paper cites.
Partialling out the spatial component of ecological variation
Daniel Borcard, Pierre Legendre, and Pierre Drapeau · 1992
Earlier work this paper cites.
Scaling laws for neural language models
Original
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei · 2001
Earlier work this paper cites.
URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin · 2002
Earlier work this paper cites.
Linguistically naïve != language independent: Why NLP needs linguistic typology
Emily M Bender · 2009
Earlier work this paper cites.
Haitian Creole language data
CMU · 2010
Earlier work this paper cites.
On achieving and evaluating language-independence in NLP
Emily M Bender · 2011
Earlier work this paper cites.
Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages
Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff · 2012
Earlier work this paper cites.
SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson · 2012
Earlier work this paper cites.
Parallel data, tools and interfaces in OPUS
Jörg Tiedemann · 2012
Earlier work this paper cites.
WALS Online (v2020.3)
Matthew S. Dryer and Martin Haspelmath (eds.) · 2013
Earlier work this paper cites.
NCHLT isiXhosa Named Entity Annotated Corpus, 2016
Kholisa Podile and Roald Eiselen · 2016
Earlier work this paper cites.
The technology of web-texts collection of Russian minor languages
Lyudmila Zaydelman, Irina Krylova, and Boris Orekhov · 2016
Earlier work this paper cites.
On the relation between linguistic typology and (limitations of) multilingual language modeling
Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen · 2018
Earlier work this paper cites.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever · 2018
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional Transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Earlier work this paper cites.
Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures
Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary · 2019
Earlier work this paper cites.
How multilingual is multilingual BERT?
Telmo Pires, Eva Schlinger, and Dan Garrette · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Earlier work this paper cites.
Energy and policy considerations for deep learning in NLP
Emma Strubell, Ananya Ganesh, and Andrew McCallum · 2019
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Earlier work this paper cites.
Unsupervised cross-lingual representation learning at scale
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov · 2020
Earlier work this paper cites.
Emerging cross-lingual structure in pretrained language models
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov · 2020
Earlier work this paper cites.
spaCy: Industrial-strength natural language processing in python
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd · 2020
Earlier work this paper cites.
The Nunavut Hansard Inuktitut–English parallel corpus 3.0 with preliminary machine translation results, 2020
Eric Joanis, Rebecca Knowles, Roland Kuhn, Samuel Larkin, Patrick Littell, Chi-kiu Lo, Darlene Stewart, and Jeffrey Micher · 2020
Earlier work this paper cites.
The state and fate of linguistic diversity and inclusion in the NLP world
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury · 2020
Earlier work this paper cites.

Similar

IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar · 2020
Cited alongside, same era.
Cross-lingual ability of multilingual BERT: An empirical study
Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth · 2020
Cited alongside, same era.
The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT
Jörg Tiedemann · 2020
Cited alongside, same era.
SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors · 2020
Cited alongside, same era.
On negative interference in multilingual models: Findings and a meta-learning treatment
Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov · 2020
Cited alongside, same era.
Transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush · 2020
Cited alongside, same era.
Are all languages created equal in multilingual BERT?
Shijie Wu and Mark Dredze · 2020
Cited alongside, same era.
ChrEn: Cherokee-English machine translation for endangered language revitalization
Shiyue Zhang, Benjamin Frey, and Mohit Bansal · 2020
Cited alongside, same era.
Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus
Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot · 2021
Cited alongside, same era.
MasakhaNER: Named entity recognition for African languages
David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei · 2021
Cited alongside, same era.
Monolingual pre-trained language models for Tigrinya
Fitsum Gaim, Wonsuk Yang, and Jong Park · 2021
Cited alongside, same era.
GELR: A bilingual Ewe-English corpus building and evaluation
Yvette Gbedevi Akouyo, Kevin Zhang, and Tchaye-Kondi Jude · 2021
Cited alongside, same era.

Then

Match the script, adapt if multilingual: Analyzing the effect of multilingual pretraining on cross-lingual transferability
Yoshinari Fujinuma, Jordan Boyd-Graber, and Katharina Kann · 2022
Later among the works it cites.
BERT syntactic transfer: A computational experiment on Italian, French and English languages
Raffaele Guarasci, Stefano Silvestri, Giuseppe De Pietro, Hamido Fujita, and Massimo Esposito · 2022
Later among the works it cites.
Improving low-resource languages in pre-trained multilingual language models
Viktor Hangya, Hossain Shaikh Saadi, and Alexander Fraser · 2022
Later among the works it cites.
Training compute-optimal large language models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre · 2022
Later among the works it cites.
Deduplicating training data makes language models better
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini · 2022
Later among the works it cites.
Few-shot learning with multilingual generative language models
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li · 2022
Later among the works it cites.
The makerere radio speech corpus: A Luganda radio corpus for automatic speech recognition
Jonathan Mukiibi, Andrew Katumba, Joyce Nakatumba-Nabende, Ali Hussein, and Joshua Meyer · 2022
Later among the works it cites.
An exploration of vocabulary size and transfer effects in multilingual language models for African languages
Akintunde Oladipo, Odunayo Ogundepo, Kelechi Ogueji, and Jimmy Lin · 2022
Later among the works it cites.
Bloom: A 176b-parameter open-access multilingual language model
Original
Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick, Suzana Ili’c, Daniel Hesslow, Roman Castagn’e, Alexandra Sasha Luccioni, Franccois Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Rose Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, et al · 2022
Later among the works it cites.
Cree corpus: A collection of nêhiyawêwin resources
Daniela Teodorescu, Josie Matalski, Delaney Lothian, Denilson Barbosa, and Carrie Demmans Epp · 2022
Later among the works it cites.
Cross-lingual few-shot learning on unseen languages
Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preotiuc-Pietro · 2022
Later among the works it cites.
Oolong: Investigating what makes crosslingual transfer hard with controlled studies
Original
Zhengxuan Wu, Isabel Papadimitriou, and Alex Tamkin · 2022
Later among the works it cites.
The Belebele benchmark: A parallel reading comprehension dataset in 122 language variants
Original
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa · 2023
Closest in time.
Fula speech corpus, 2023
Cawoylel · 2023
Closest in time.
Cherokee corpus and Cherokee-English Dictionary, 2023
Cherokee Corpus · 2023
Closest in time.
eBible, 2023
eBible · 2023
Closest in time.
Zero-shot cross-lingual transfer language selection using linguistic similarity
Juuso Eronen, Michal Ptaszynski, and Fumito Masui · 2023
Closest in time.
Glottolog 4.8
Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank · 2023
Closest in time.
Glot500: Scaling multilingual corpora and language models to 500 languages
Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze · 2023
Closest in time.
Advanced Multivariate Analyses in R: Variation Partitioning
QCBS · 2023
Closest in time.
Ulukau: The Hawaiian Electronic Library
Ulukau · 2023
Closest in time.
Wikimedia dumps, 2023
Wikimedia · 2023
Closest in time.
Wikipedia, 2023
Wikipedia · 2023
Closest in time.
NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder · 2023
Closest in time.
Training trajectories of language models across scales
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Veselin Stoyanov · 2023
Closest in time.
How good are typological distances for determining genealogical relationships among languages?
Taraka Rama and Prasanth Kolachina · 2095
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…