2024

MaLA-500: Massive Language Adaptation of Large Language Models

Lin, Peiqin, Ji, Shaoxiong, Tiedemann, Jörg et al.

Understand

Large language models (LLMs) have advanced the state of the art in natural language processing.

However, their predominant design for English or a limited set of languages creates a substantial gap in their effectiveness for low-resource languages.
To bridge this gap, we introduce MaLA-500, a novel large language model designed to cover an extensive range of 534 languages.
To train MaLA-500, we employ vocabulary extension and continued pretraining on LLaMA 2 with Glot500-c.

Built on

Can multilingual language models transfer to an unseen dialect? A case study on north african arabizi
Original
Benjamin Müller, Benoît Sagot, and Djamé Seddah · 2005
Earlier work this paper cites.
SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson · 2012
Earlier work this paper cites.
Creating a massively parallel bible corpus
Thomas Mayer and Michael Cysouw · 2014
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Earlier work this paper cites.
Parsing with multilingual bert, a small treebank, and a small corpus
Ethan C. Chau, Lucy H. Lin, and Noah A. Smith · 2020
Earlier work this paper cites.
ELECTRA: Pre-training text encoders as discriminators rather than generators
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning · 2020
Earlier work this paper cites.
Unsupervised cross-lingual representation learning at scale
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov · 2020
Earlier work this paper cites.
Multilingual denoising pre-training for neural machine translation
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer · 2020
Earlier work this paper cites.
MAD-X: an adapter-based framework for multi-task cross-lingual transfer
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder · 2020
Earlier work this paper cites.
ZeRO: Memory optimizations toward training trillion parameter models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He · 2020
Earlier work this paper cites.
DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He · 2020
Earlier work this paper cites.
Simalign: High quality word alignments without parallel training data using static and contextualized embeddings
Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze · 2020
Earlier work this paper cites.
Udapter: Language adaptation for truly universal dependency parsing
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord · 2020
Earlier work this paper cites.
Extending multilingual BERT to low-resource languages
Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth · 2020
Earlier work this paper cites.
Transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al · 2020
Earlier work this paper cites.
How to adapt your pretrained multilingual model to 1600 languages
Abteen Ebrahimi and Katharina Kann · 2021
Earlier work this paper cites.
Few-shot learning with multilingual language models
Original
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li · 2021
Earlier work this paper cites.

Similar

When being unseen from mbert is just the beginning: Handling new languages with multilingual language models
Benjamin Müller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah · 2021
Cited alongside, same era.
Trankit: A light-weight transformer-based toolkit for multilingual natural language processing
Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen · 2021
Cited alongside, same era.
Unks everywhere: Adapting multilingual language models to new scripts
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder · 2021
Cited alongside, same era.
mT5: A massively multilingual pre-trained text-to-text transformer
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel · 2021
Cited alongside, same era.
SERENGETI: Massively multilingual language models for africa
Original
Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte · 2022
Cited alongside, same era.
Adapting pre-trained language models to african languages via multilingual adaptive fine-tuning
Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow · 2022
Cited alongside, same era.
Phylogeny-inspired adaptation of multilingual models to new languages
Original
Fahim Faisal and Antonios Anastasopoulos · 2022
Cited alongside, same era.
LoRA: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen · 2022
Cited alongside, same era.
What makes good in-context examples for gpt-3?
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen · 2022
Cited alongside, same era.
BLOOM: A 176b-parameter open-access multilingual language model
Original
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al · 2022
Cited alongside, same era.
mGPT: Few-shot learners go multilingual
Original
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina · 2022
Cited alongside, same era.
UL2: Unifying language learning paradigms
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al · 2022
Cited alongside, same era.

Then

A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity
Original
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung · 2023
Later among the works it cites.
Instruct-align: Teaching novel languages with to LLMs through alignment-based cross-lingual instruction
Original
Samuel Cahyawijaya, Holy Lovenia, Tiezheng Yu, Willy Chung, and Pascale Fung · 2023
Later among the works it cites.
Improving language plasticity via pretraining with active forgetting
Original
Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifeoluwa Adelani, Pontus Stenetorp, Sebastian Riedel, and Mikel Artetxe · 2023
Later among the works it cites.
Efficient and effective text encoding for Chinese LLaMA and Alpaca
Original
Yiming Cui, Ziqing Yang, and Xin Yao · 2023
Later among the works it cites.
Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages
Original
C. M. Downey, Terra Blevins, Nora Goldfine, and Shane Steinert-Threlkeld · 2023
Later among the works it cites.
Glot500: Scaling multilingual corpora and language models to 500 languages
Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze · 2023
Later among the works it cites.
Mistral 7b
Original
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed · 2023
Later among the works it cites.
MADLAD-400: A multilingual and document-level large audited dataset
Original
Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, et al · 2023
Later among the works it cites.
ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning
Original
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen · 2023
Later among the works it cites.
Taxi1500: A multilingual dataset for text classification in 1500 languages, 2023
Chunlan Ma, Ayyoob ImaniGooghari, Haotian Ye, Ehsaneddin Asgari, and Hinrich Schütze · 2023
Later among the works it cites.
A paradigm shift in machine translation: Boosting translation performance of large language models
Original
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla · 2023
Later among the works it cites.
Bigtrans: Augmenting large language models with multilingual translation capability over 100 languages
Original
Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong · 2023
Later among the works it cites.
MEGABYTE: predicting million-byte sequences with multiscale transformers
Original
Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis · 2023
Later among the works it cites.
Extrapolating large language models to non-english by aligning languages
Original
Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li · 2023
Later among the works it cites.
Monolingual or multilingual instruction tuning: Which makes a better Alpaca
Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Barry Haddow, and Kenneth Heafield · 2024
Closest in time.
Mixtral of experts
Original
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al · 2024
Closest in time.
LLaMA beyond English: An empirical study on language capability transfer
Original
Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…