Character, Word, or Both? Revisiting the Segmentation Granularity for Chinese Pre-trained Language Models

Understand

Pretrained language models (PLMs) have shown marvelous improvements across various NLP tasks.

• Most Chinese PLMs simply treat an input text as a sequence of characters, and completely ignore word information.
• Although Whole Word Masking can alleviate this, the semantics in words is still not well represented.
• In this paper, we revisit the segmentation granularity of Chinese PLMs.