General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference

Understand

The state of the art on many NLP tasks is currently achieved by large pre-trained language models, which require a considerable amount of computation.

• We explore a setting where many different predictions are made on a single piece of text.
• In that case, some of the computational cost during inference can be amortized over the different tasks using a shared text encoder.
• We compare approaches for training such an encoder and show that encoders pre-trained over multiple tasks generalize well to unseen tasks.