Causal Estimation of Tokenisation Bias

Understand

Modern language models are typically trained over subword sequences, but ultimately define probabilities over character-strings.

• Ideally, the choice of the tokeniser -- which maps character-strings to subwords -- should not affect the probability assigned to the underlying character-string; in practice, it does.
• We define this mismatch as tokenisation bias.
• In this work, we quantify one particular type of tokenisation bias: the effect of including or not a subword (e.g., $\langle hello \rangle$) in a tokeniser's vocabulary on the probability a trained model assigns to the corresponding characters (i.e., \textit{``hello''}).