Autocorrelations Decay in Texts and Applicability Limits of Language Models

Understand

We show that the laws of autocorrelations decay in texts are closely related to applicability limits of language models.

• Using distributional semantics we empirically demonstrate that autocorrelations of words in texts decay according to a power law.
• We show that distributional semantics provides coherent autocorrelations decay exponents for texts translated to multiple languages.
• The autocorrelations decay in generated texts is quantitatively and often qualitatively different from the literary texts.