Coherence boosting: When your pretrained language model is not paying enough attention

Understand

Long-range semantic coherence remains a challenge in automatic language generation and understanding.

• We demonstrate that large language models have insufficiently learned the effect of distant words on next-token prediction.
• We present coherence boosting, an inference procedure that increases a LM's focus on a long context.
• We show the benefits of coherence boosting with pretrained models by distributional analyses of generated ordinary text and dialog responses.