Sentence Bottleneck Autoencoders from Transformer Language Models

Understand

Representation learning for text via pretraining a language model on a large corpus has become a standard starting point for building NLP systems.

• This approach stands in contrast to autoencoders, also trained on raw text, but with the objective of learning to encode each input as a vector that allows full reconstruction.
• Autoencoders are attractive because of their latent space structure and generative properties.
• We therefore explore the construction of a sentence-level autoencoder from a pretrained, frozen transformer language model.