Phase transition in large language models and the criticality of natural languages

Understand

Generation of text and speech in natural languages can be modeled as a stochastic process.

• This idea dates back to the seminal work of Markov and, later, to that of Shannon and also underlies the recent development of large language models (LLMs).
• The stochastic processes corresponding to natural languages should be distinct from those that generate nonlinguistic sequences.
• One of the features that discriminate linguistic and nonlinguistic sequences is power-law behavior, which is universally observed across different languages.