Multiscale sequence modeling with a learned dictionary

Understand

We propose a generalization of neural network sequence models.

• Instead of predicting one symbol at a time, our multi-scale model makes predictions over multiple, potentially overlapping multi-symbol tokens.
• A variation of the byte-pair encoding (BPE) compression algorithm is used to learn the dictionary of tokens that the model is trained with.
• When applied to language modelling, our model has the flexibility of character-level models while maintaining many of the performance benefits of word-level models.