2024

LLaMA Beyond English: An Empirical Study on Language Capability Transfer

Zhao, Jun, Zhang, Zhihao, Gao, Luhui et al.

Understand

In recent times, substantial advancements have been witnessed in large language models (LLMs), exemplified by ChatGPT, showcasing remarkable proficiency across a range of complex tasks.

However, many mainstream LLMs (e.g.
LLaMA) are pretrained on English-dominant corpus, which limits their performance in other non-English languages.
In this paper, we focus on how to effectively transfer the capabilities of language generation and following instructions to a non-English language.

Built on

Unsupervised Cross-lingual Representation Learning at Scale
Original
Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; and Stoyanov, V. 2020a · 1911
Earlier work this paper cites.
Multilingual Alignment of Contextual Word Representations
Original
Cao, S.; Kitaev, N.; and Klein, D. 2020 · 2002
Earlier work this paper cites.
Measuring Massive Multitask Language Understanding
Original
Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020 · 2009
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019 · 2019
Earlier work this paper cites.
How Multilingual is Multilingual BERT?
Pires, T.; Schlinger, E.; and Garrette, D. 2019 · 2019
Earlier work this paper cites.
Learning Multilingual Meta-Embeddings for Code-Switching Named Entity Recognition
Winata, G. I.; Lin, Z.; and Fung, P. 2019 · 2019
Earlier work this paper cites.
Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition
Winata, G. I.; Lin, Z.; Shin, J.; Liu, Z.; and Fung, P. 2019 · 2019
Earlier work this paper cites.
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
Wu, S.; and Dredze, M. 2019 · 2019
Earlier work this paper cites.
On the Cross-lingual Transferability of Monolingual Representations
Artetxe, M.; Ruder, S.; and Yogatama, D. 2020 · 2020
Earlier work this paper cites.
Finding Universal Grammatical Relations in Multilingual BERT
Chi, E. A.; Hewitt, J.; and Manning, C. D. 2020 · 2020
Earlier work this paper cites.
Identifying Elements Essential for BERT’s Multilinguality
Dufter, P.; and Schütze, H. 2020 · 2020
Earlier work this paper cites.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Jiang, Z.; Anastasopoulos, A.; Araki, J.; Ding, H.; and Neubig, G. 2020 · 2020
Earlier work this paper cites.
The State and Fate of Linguistic Diversity and Inclusion in the NLP World
Joshi, P.; Santy, S.; Budhiraja, A.; Bali, K.; and Choudhury, M. 2020 · 2020
Earlier work this paper cites.
GLUECoS: An Evaluation Benchmark for Code-Switched NLP
Khanuja, S.; Dandapat, S.; Srinivasan, A.; Sitaram, S.; and Choudhury, M. 2020 · 2020
Earlier work this paper cites.

Similar

Training Verifiers to Solve Math Word Problems
Original
Cobbe, K.; Kosaraju, V.; Bavarian, M.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021 · 2021
Cited alongside, same era.
LoRA: Low-Rank Adaptation of Large Language Models
Original
Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W. 2021 · 2021
Cited alongside, same era.
Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Filling
Original
Krishnan, J.; Anastasopoulos, A.; Purohit, H.; and Rangwala, H. 2021 · 2021
Cited alongside, same era.
Linguistic Diversity in Natural Language Processing
Ranta, A.; and Goutte, C. 2021 · 2021
Cited alongside, same era.
Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots
Original
Tan, S.; and Joty, S. 2021 · 2021
Cited alongside, same era.
Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
Original
Chen, G.; Ma, S.; Chen, Y.; Zhang, D.; Pan, J.; Wang, W.; and Wei, F. 2022 · 2022
Cited alongside, same era.
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Huang, W.; Abbeel, P.; Pathak, D.; and Mordatch, I. 2022 · 2022
Cited alongside, same era.
Few-shot Learning with Multilingual Language Models
Original
Lin, X. V.; Mihaylov, T.; Artetxe, M.; Wang, T.; Chen, S.; Simig, D.; Ott, M.; Goyal, N.; Bhosale, S.; Du, J.; Pasunuru, R.; Shleifer, S.; Koura, P. S.; Chaudhary, V.; O’Horo, B.; Wang, J.; Zettlemoyer, L.; Kozareva, Z.; Diab, M.; Stoyanov, V.; and Li, X. 2022 · 2022
Cited alongside, same era.
Introducing ChatGPT
OpenAI. 2022 · 2022
Cited alongside, same era.
Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies
Original
Wu, Z.; Papadimitriou, I.; and Tamkin, A. 2022 · 2022
Cited alongside, same era.
PaLM 2 Technical Report
Original
Anil, R.; Dai, A. M.; Firat, O.; Johnson, M.; and Lepikhin, D. 2023 · 2023
Cited alongside, same era.
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Original
Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; Nori, H.; Palangi, H.; Ribeiro, M. T.; and Zhang, Y. 2023 · 2023
Cited alongside, same era.

Then

A Survey on In-context Learning
Original
Dong, Q.; Li, L.; Dai, D.; Zheng, C.; Wu, Z.; Chang, B.; Sun, X.; Xu, J.; Li, L.; and Sui, Z. 2023 · 2023
Later among the works it cites.
Zero-shot cross-lingual transfer language selection using linguistic similarity
Eronen, J.; Ptaszynski, M.; and Masui, F. 2023 · 2023
Later among the works it cites.
BELLE: Be Everyone’s Large Language model Engine
Ji, Y.; Deng, Y.; Gong, Y.; Peng, Y.; Niu, Q.; Ma, B.; and Li, X. 2023 · 2023
Later among the works it cites.
Gpt-4 passes the bar exam
Katz, D. M.; Bommarito, M. J.; Gao, S.; and Arredondo, P. 2023 · 2023
Later among the works it cites.
Bactrian-X : A Multilingual Replicable Instruction-Following Model with Low-Rank Adaptation
Original
Li, H.; Koto, F.; Wu, M.; Aji, A. F.; and Baldwin, T. 2023 · 2023
Later among the works it cites.
Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts
Original
Nguyen, X.-P.; Aljunied, S. M.; Joty, S.; and Bing, L. 2023 · 2023
Later among the works it cites.
Open-Chinese-LLaMA
OpenLMLab. 2023 · 2023
Later among the works it cites.
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
Original
Penedo, G.; Malartic, Q.; Hesslow, D.; Cojocaru, R.; Cappelli, A.; Alobeidli, H.; Pannier, B.; Almazrouei, E.; and Launay, J. 2023 · 2023
Later among the works it cites.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Original
Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; Ilić, S.; Hesslow, D.; and Castagné, R. 2023 · 2023
Later among the works it cites.
Announcing StableCode
StabilityAI. 2023 · 2023
Later among the works it cites.
Alpaca: A Strong, Replicable Instruction-Following Model
Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023 · 2023
Later among the works it cites.
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Original
Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2023 · 2023
Later among the works it cites.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Original
Zhu, W.; Liu, H.; Dong, Q.; Xu, J.; Huang, S.; Kong, L.; Chen, J.; and Li, L. 2023 · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…