A Simple Contrastive Learning Objective for Alleviating Neural Text Degeneration

Understand

The cross-entropy objective has proved to be an all-purpose training objective for autoregressive language models (LMs).

• However, without considering the penalization of problematic tokens, LMs trained using cross-entropy exhibit text degeneration.
• To address this, unlikelihood training has been proposed to reduce the probability of unlikely tokens predicted by LMs.
• But unlikelihood does not consider the relationship between the label tokens and unlikely token candidates, thus showing marginal improvements in degeneration.