Multi-timescale Representation Learning in LSTM Language Models

Understand

Language models must capture statistical dependencies between words at timescales ranging from very short to very long.

• Earlier work has demonstrated that dependencies in natural language tend to decay with distance between words according to a power law.
• However, it is unclear how this knowledge can be used for analyzing or designing neural network language models.
• In this work, we derived a theory for how the memory gating mechanism in long short-term memory (LSTM) language models can capture power law decay.