Language-Independent Tokenisation Rivals Language-Specific Tokenisation for Word Similarity Prediction

Understand

Language-independent tokenisation (LIT) methods that do not require labelled language resources or lexicons have recently gained popularity because of their applicability in resource-poor languages.

• Moreover, they compactly represent a language using a fixed size vocabulary and can efficiently handle unseen or rare words.
• On the other hand, language-specific tokenisation (LST) methods have a long and established history, and are developed using carefully created lexicons and training resources.
• Unlike subtokens produced by LIT methods, LST methods produce valid morphological subwords.