Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

Understand

State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings.

• In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model.
• To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion.
• Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network.