MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing

LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

Dou, S.; Zhou, E.; Liu, Y.; Gao, S.; Shen, W.; Xiong, L.; Zhou, Y.; Wang, X.; Xi, Z.; Fan, X.; Pu, S.; Zhu, J.; Zheng, R.; Gui, T.; Zhang, Q.; and Huang, X. 2024 · 1945

Earlier work this paper cites.

Scaling laws for neural language models

Original

Kaplan, J.; McCandlish, S.; Henighan, T.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D. 2020 · 2001

Earlier work this paper cites.

Think you have solved question answering? try arc, the ai2 reasoning challenge

Original

Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018 · 2018

Earlier work this paper cites.

HellaSwag: Can a Machine Really Finish Your Sentence?

Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019 · 2019

Earlier work this paper cites.

Measuring Massive Multitask Language Understanding

Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020 · 2020

Earlier work this paper cites.

Are All Languages Created Equal in Multilingual BERT?

Wu, S.; and Dredze, M. 2020 · 2020

Earlier work this paper cites.

LoRA: Low-Rank Adaptation of Large Language Models

Hu, E. J.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2021 · 2021

Earlier work this paper cites.

No language left behind: Scaling human-centered machine translation

Original

Costa-jussà, M. R.; Cross, J.; Çelebi, O.; Elbayad, M.; Heafield, K.; Heffernan, K.; Kalbassi, E.; Lam, J.; Licht, D.; Maillard, J.; et al. 2022 · 2022

Earlier work this paper cites.

Flashattention: Fast and memory-efficient exact attention with io-awareness

Dao, T.; Fu, D.; Ermon, S.; Rudra, A.; and Ré, C. 2022 · 2022

Earlier work this paper cites.

Glam: Efficient scaling of language models with mixture-of-experts

Du, N.; Huang, Y.; Dai, A. M.; Tong, S.; Lepikhin, D.; Xu, Y.; Krikun, M.; Zhou, Y.; Yu, A. W.; Firat, O.; et al. 2022 · 2022

Earlier work this paper cites.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

Fedus, W.; Zoph, B.; and Shazeer, N. 2022 · 2022

Earlier work this paper cites.

The flores-101 evaluation benchmark for low-resource and multilingual machine translation

Goyal, N.; Gao, C.; Chaudhary, V.; Chen, P.-J.; Wenzek, G.; Ju, D.; Krishnan, S.; Ranzato, M.; Guzmán, F.; and Fan, A. 2022 · 2022

Earlier work this paper cites.

An empirical analysis of compute-optimal large language model training

Hoffmann, J.; Borgeaud, S.; Mensch, A.; Buchatskaya, E.; Cai, T.; Rutherford, E.; de Las Casas, D.; Hendricks, L. A.; Welbl, J.; Clark, A.; et al. 2022 · 2022

Earlier work this paper cites.

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

Komatsuzaki, A.; Puigcerver, J.; Lee-Thorp, J.; Ruiz, C. R.; Mustafa, B.; Ainslie, J.; Tay, Y.; Dehghani, M.; and Houlsby, N. 2022 · 2022

Earlier work this paper cites.

COMET-22: Unbabel-IST 2022 submission for the metrics shared task

Rei, R.; De Souza, J. G.; Alves, D.; Zerva, C.; Farinha, A. C.; Glushkova, T.; Lavie, A.; Coheur, L.; and Martins, A. F. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023 · 2023

Earlier work this paper cites.

Qwen technical report

Original

Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023 · 2023

Cited alongside, same era.

The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants

Original

Bandarkar, L.; Liang, D.; Muller, B.; Artetxe, M.; Shukla, S. N.; Husa, D.; Goyal, N.; Krishnan, A.; Zettlemoyer, L.; and Khabsa, M. 2023 · 2023

Cited alongside, same era.

Scaling expert language models with unsupervised domain discovery

Original

Gururangan, S.; Li, M.; Lewis, M.; Shi, W.; Althoff, T.; Smith, N. A.; and Zettlemoyer, L. 2023 · 2023

Cited alongside, same era.

Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback

Lai, V.; Nguyen, C.; Ngo, N.; Nguyen, T.; Dernoncourt, F.; Rossi, R.; and Nguyen, T. 2023 · 2023

Cited alongside, same era.

Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models

Original

Blevins, T.; Limisiewicz, T.; Gururangan, S.; Li, M.; Gonen, H.; Smith, N. A.; and Zettlemoyer, L. 2024 · 2024

Closest in time.

Sambalingo: Teaching large language models new languages

Original

Csaki, Z.; Li, B.; Li, J.; Xu, Q.; Pawakapan, P.; Zhang, L.; Du, Y.; Zhao, H.; Hu, C.; and Thakker, U. 2024 · 2024

Closest in time.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Original

DeepSeek-AI. 2024 · 2024

Closest in time.

The llama 3 herd of models

Original

Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024 · 2024

Closest in time.