Improving Tokenisation by Alternative Treatment of Spaces

Understand

Tokenisation is the first step in almost all NLP tasks, and state-of-the-art transformer-based language models all use subword tokenisation algorithms to process input text.

• Existing algorithms have problems, often producing tokenisations of limited linguistic validity, and representing equivalent strings differently depending on their position within a word.
• We hypothesise that these problems hinder the ability of transformer-based models to handle complex words, and suggest that these problems are a result of allowing tokens to include spaces.
• We thus experiment with an alternative tokenisation approach where spaces are always treated as individual tokens.