On the Limits of Language Generation: Trade-Offs Between Hallucination and Mode Collapse

Understand

Specifying all desirable properties of a language model is challenging, but certain requirements seem essential.

• Given samples from an unknown language, the trained model should produce valid strings not seen in training and be expressive enough to capture the language's full richness.
• Otherwise, outputting invalid strings constitutes "hallucination," and failing to capture the full range leads to "mode collapse." We ask if a language model can meet both requirements.
• We investigate this within a statistical language generation setting building on Gold and Angluin.