The word entropy of natural languages

Understand

The average uncertainty associated with words is an information-theoretic concept at the heart of quantitative and computational linguistics.

• The entropy has been established as a measure of this average uncertainty - also called average information content.
• We here use parallel texts of 21 languages to establish the number of tokens at which word entropies converge to stable values.
• These convergence points are then used to select texts from a massively parallel corpus, and to estimate word entropies across more than 1000 languages.