Language as a Latent Variable: Discrete Generative Models for Sentence Compression

Understand

In this work we explore deep generative models of text in which the latent representation of a document is itself drawn from a discrete language model distribution.

• We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences.
• In this application the generative model first draws a latent summary sentence from a background language model, and then subsequently draws the observed sentence conditioned on this latent summary.
• In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data.