Sub-Character Tokenization for Chinese Pretrained Language Models

Understand

Tokenization is fundamental to pretrained language models (PLMs).

• Existing tokenization methods for Chinese PLMs typically treat each character as an indivisible token.
• However, they ignore the unique feature of the Chinese writing system where additional linguistic information exists below the character level, i.e., at the sub-character level.
• To utilize such information, we propose sub-character (SubChar for short) tokenization.