Text and Code Embeddings by Contrastive Pre-Training

Understand

Text embeddings are useful features in many applications such as semantic search and computing text similarity.

• Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture.
• In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code.
• The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models.