Language Model Tokenizers Introduce Unfairness Between Languages

Understand

Recent language models have shown impressive multilingual performance, even when not explicitly trained for it.

• Despite this, there are concerns about the quality of their outputs across different languages.
• In this paper, we show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked.
• The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.