Built on

S. Lakew, et al, ”Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary,” in Proceedings of the 15th International Conference on Spoken Language Translation, 2018, pp. 54–61
2018
Earlier work this paper cites.
W. Yoon, et al, ”Pre-trained language model for biomedical question answering,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2019, pp. 727–740
2019
Earlier work this paper cites.
J. Devlin, et al, ”BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186
2019
Earlier work this paper cites.
Kuratov Y., Arkhipov M. Adaptation of deep bidirectional multilingual transformers for Russian language //arXiv preprint arXiv:1905.07213. – 2019
Original
2019
Earlier work this paper cites.
T. Brown, et al, ”Language Models are Few-Shot Learners” //arXiv preprint arXiv:2005.14165. – 2020
Original
2020
Earlier work this paper cites.
J. Zhang, et al, ”PEGASUS: Pre-Training with Extracted Gap-Sentences for Abstractive Summarization,” in Proceedings of the 37th International Conference on Machine Learning, 2020
2020
Earlier work this paper cites.

Similar

M. Tikhomirov, et al, ”Using bert and augmentation in named entity recognition for cybersecurity domain,” in Natural Language Processing and Information Systems: 25th International Conference on Applications of Natural Language to Information Systems, NLDB 2020, Saarbrucken, Germany, June 24–26, 2020, Proceedings 25, 2020, pp. 16–24
2020
Cited alongside, same era.
K. Bostrom, G. Durrett, ”Byte Pair Encoding is Suboptimal for Language Model Pretraining,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 4617–4624
2020
Cited alongside, same era.
A. Nayak, et al, ”Domain adaptation challenges of BERT in tokenization and sub-word representations of Out-of-Vocabulary words,” in Proceedings of the First Workshop on Insights from Negative Results in NLP, 2020, pp. 1–5
2020
Cited alongside, same era.
T. Shavrina, et al, ”RussianSuperGLUE: A Russian language understanding evaluation benchmark” //arXiv preprint arXiv:2010.15925. – 2020
Original

Then

W. Vries, M. Nissim, ”As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages,” in Findings of the Association for Computational Linguistics: ACL-ĲCNLP 2021, 2021, pp. 836–846
2021
Later among the works it cites.
Hu E. J. et al. Lora: Low-rank adaptation of large language models //arXiv preprint arXiv:2106.09685. – 2021
Original
2021
Later among the works it cites.
Q. Zeng, et al, ”GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost” //arXiv preprint arXiv:2211.06993. – 2022
Original
2022
Later among the works it cites.
L. Ouyang, et al, ”Training language models to follow instructions with human feedback” //arXiv preprint arXiv:2203.02155. – 2022
Original
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

Impact of Tokenization on LLaMa Russian Adaptation

Tikhomirov, Mikhail, Chernyshev, Daniil

Understand

Latest instruction-tuned large language models (LLM) show great results on various tasks, however, they often face performance degradation for non-English input.

There is evidence that the reason lies in inefficient tokenization caused by low language representation in pre-training data which hinders the comprehension of non-English instructions, limiting the potential of target language instruction-tuning.
In this work we investigate the possibility of addressing the issue with vocabulary substitution in the context of LLaMa Russian language adaptation.
We explore three variants of vocabulary adaptation and test their performance on Saiga instruction-tuning and fine-tuning on Russian Super Glue benchmark.

Built on

S. Lakew, et al, ”Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary,” in Proceedings of the 15th International Conference on Spoken Language Translation, 2018, pp. 54–61
2018
Earlier work this paper cites.
W. Yoon, et al, ”Pre-trained language model for biomedical question answering,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2019, pp. 727–740
2019
Earlier work this paper cites.
J. Devlin, et al, ”BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186
2019
Earlier work this paper cites.
Kuratov Y., Arkhipov M. Adaptation of deep bidirectional multilingual transformers for Russian language //arXiv preprint arXiv:1905.07213. – 2019
Original
2019
Earlier work this paper cites.
T. Brown, et al, ”Language Models are Few-Shot Learners” //arXiv preprint arXiv:2005.14165. – 2020
Original
2020
Earlier work this paper cites.
J. Zhang, et al, ”PEGASUS: Pre-Training with Extracted Gap-Sentences for Abstractive Summarization,” in Proceedings of the 37th International Conference on Machine Learning, 2020
2020
Earlier work this paper cites.

Similar

M. Tikhomirov, et al, ”Using bert and augmentation in named entity recognition for cybersecurity domain,” in Natural Language Processing and Information Systems: 25th International Conference on Applications of Natural Language to Information Systems, NLDB 2020, Saarbrucken, Germany, June 24–26, 2020, Proceedings 25, 2020, pp. 16–24
2020
Cited alongside, same era.
K. Bostrom, G. Durrett, ”Byte Pair Encoding is Suboptimal for Language Model Pretraining,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 4617–4624
2020
Cited alongside, same era.
A. Nayak, et al, ”Domain adaptation challenges of BERT in tokenization and sub-word representations of Out-of-Vocabulary words,” in Proceedings of the First Workshop on Insights from Negative Results in NLP, 2020, pp. 1–5
2020
Cited alongside, same era.
T. Shavrina, et al, ”RussianSuperGLUE: A Russian language understanding evaluation benchmark” //arXiv preprint arXiv:2010.15925. – 2020
Original

Then

W. Vries, M. Nissim, ”As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages,” in Findings of the Association for Computational Linguistics: ACL-ĲCNLP 2021, 2021, pp. 836–846
2021
Later among the works it cites.
Hu E. J. et al. Lora: Low-rank adaptation of large language models //arXiv preprint arXiv:2106.09685. – 2021
Original
2021
Later among the works it cites.
Q. Zeng, et al, ”GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost” //arXiv preprint arXiv:2211.06993. – 2022
Original
2022
Later among the works it cites.
L. Ouyang, et al, ”Training language models to follow instructions with human feedback” //arXiv preprint arXiv:2203.02155. – 2022
Original
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2020

Cited alongside, same era.

P. Rust, et al, ”How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 3118–3135

2021

Cited alongside, same era.

V. Hofmann, J. Pierrehumbert, H. Schutze, ”Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 3594–3608

2021

Cited alongside, same era.

Y. Cui, et al, ”Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca” //arXiv preprint arXiv:2304.08177. – 2023

Original

2023

Closest in time.

H. Touvron, et al, ”LLaMA: Open and Efficient Foundation Language Models” //arXiv preprint arXiv:2302.13971. – 2023

Original

2023

Closest in time.