ColdGANs: Taming Language GANs with Cautious Sampling Strategies

Understand

Training regimes based on Maximum Likelihood Estimation (MLE) suffer from known limitations, often leading to poorly generated text sequences.

• At the root of these limitations is the mismatch between training and inference, i.e.
• the so-called exposure bias, exacerbated by considering only the reference texts as correct, while in practice several alternative formulations could be as good.
• Generative Adversarial Networks (GANs) can mitigate those limitations but the discrete nature of text has hindered their application to language generation: the approaches proposed so far, based on Reinforcement Learning, have been shown to underperform MLE.