Lexically Grounded Subword Segmentation

Understand

We present three innovations in tokenization and subword segmentation.

• First, we propose to use unsupervised morphological analysis with Morfessor as pre-tokenization.
• Second, we present an algebraic method for obtaining subword embeddings grounded in a word embedding space.
• Based on that, we design a novel subword segmentation algorithm that uses the embeddings, ensuring that the procedure considers lexical meaning.