Compressing Neural Language Models by Sparse Word Representations

Understand

Neural networks are among the state-of-the-art techniques for language modeling.

• Existing neural language models typically map discrete words to distributed, dense vector representations.
• After information processing of the preceding context words by hidden layers, an output layer estimates the probability of the next word.
• Such approaches are time- and memory-intensive because of the large numbers of parameters for word embeddings and the output layer.