WEST: Word Encoded Sequence Transducers

Understand

Most of the parameters in large vocabulary models are used in embedding layer to map categorical features to vectors and in softmax layer for classification weights.

• This is a bottle-neck in memory constraint on-device training applications like federated learning and on-device inference applications like automatic speech recognition (ASR).
• One way of compressing the embedding and softmax layers is to substitute larger units such as words with smaller sub-units such as characters.
• However, often the sub-unit models perform poorly compared to the larger unit models.