Optimizing Segmentation Granularity for Neural Machine Translation

Understand

In neural machine translation (NMT), it is has become standard to translate using subword units to allow for an open vocabulary and improve accuracy on infrequent words.

• Byte-pair encoding (BPE) and its variants are the predominant approach to generating these subwords, as they are unsupervised, resource-free, and empirically effective.
• However, the granularity of these subword units is a hyperparameter to be tuned for each language and task, using methods such as grid search.
• Tuning may be done inexhaustively or skipped entirely due to resource constraints, leading to sub-optimal performance.