Enabling Language Models to Fill in the Blanks

Understand

We present a simple approach for text infilling, the task of predicting missing spans of text at any position in a document.

• While infilling could enable rich functionality especially for writing assistance tools, more attention has been devoted to language modeling---a special case of infilling where text is predicted at the end of a document.
• In this paper, we aim to extend the capabilities of language models (LMs) to the more general task of infilling.
• To this end, we train (or fine-tune) off-the-shelf LMs on sequences containing the concatenation of artificially-masked text and the text which was masked.