Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

Understand

We provide the first exploration of sentence embeddings from text-to-text transformers (T5).

• Sentence embeddings are broadly useful for language processing tasks.
• While T5 achieves impressive performance on language tasks cast as sequence-to-sequence mapping problems, it is unclear how to produce sentence embeddings from encoder-decoder models.
• We investigate three methods for extracting T5 sentence embeddings: two utilize only the T5 encoder and one uses the full T5 encoder-decoder model.