Fetching the paper…

Reading the bibliography…

2023

Training and Evaluation of a Multilingual Tokenizer for GPT-SW3

Stollenwerk, Felix

Original
PDF
Post on X

Understand

This paper provides a detailed discussion of the multilingual tokenizer used for GPT-SW3.

It was trained on the Nordic Pile using the SentencePiece library and the BPE algorithm.
We outline the tokenizer's most important features and share details on its learned vocabulary.
In addition, we systematically analyze the properties and evaluate the performance of the tokenizer with regard to the different languages present in the data.

Training and Evaluation of a Multilingual Tokenizer for GPT-SW3 · Around

Built on

“Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism”, 2020
Original
Mohammad Shoeybi et al · 1909
Earlier work this paper cites.
“SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing”
Taku Kudo and John Richardson · 2012
Earlier work this paper cites.
“Neural Machine Translation of Rare Words with Subword Units”, 2016
Original
Rico Sennrich, Barry Haddow and Alexandra Birch · 2016
Earlier work this paper cites.

Similar

“Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates”, 2018
Original
Taku Kudo · 2018
Cited alongside, same era.
“Exploring BERT’s Vocabulary”, 2019
Judit Acs · 2019
Cited alongside, same era.
“How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models”
Phillip Rust et al · 2021
Cited alongside, same era.
“GPT-SW3: An Autoregressive Language Model for the Nordic Languages” in preparation
Original
Ariel Ekgren et al
Cited in the paper.

Then

“GPT-NeoX-20B: An Open-Source Autoregressive Language Model”, 2022
Original
Sid Black et al · 2022
Later among the works it cites.
“The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling”, 2023
Original
Joey Öhman et al · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…