POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training

Understand

Large-scale pre-trained language models, such as BERT and GPT-2, have achieved excellent performance in language representation learning and free-form text generation.

• However, these models cannot be directly employed to generate text under specified lexical constraints.
• To address this challenge, we present POINTER (PrOgressive INsertion-based TransformER), a simple yet novel insertion-based approach for hard-constrained text generation.
• The proposed method operates by progressively inserting new tokens between existing tokens in a parallel manner.