Language modeling via stochastic processes

Understand

Modern language models can generate high-quality short texts.

• However, they often meander or are incoherent when generating longer texts.
• These issues arise from the next-token-only language modeling objective.
• Recent work in self-supervised learning suggests that models can learn good latent representations via contrastive learning, which can be effective for discriminative tasks.