Strategies for Training Large Vocabulary Neural Language Models

Understand

Training neural network language models over large vocabularies is still computationally very costly compared to count-based models such as Kneser-Ney.

• At the same time, neural language models are gaining popularity for many applications such as speech recognition and machine translation whose success depends on scalability.
• We present a systematic comparison of strategies to represent and train large vocabularies, including softmax, hierarchical softmax, target sampling, noise contrastive estimation and self normalization.
• We further extend self normalization to be a proper estimator of likelihood and introduce an efficient variant of softmax.