Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddings

Understand

The advancement of transformer neural networks has significantly elevated the capabilities of sentence similarity models, but they still struggle with highly discriminative tasks and may produce sub-optimal representations of important documents like scientific literature.

• With the increased reliance on retrieval augmentation and search, representing diverse documents as concise and descriptive vectors is crucial.
• This paper improves upon the vectors embeddings of scientific text by assembling niche datasets using co-citations as a similarity metric, focusing on biomedical domains.
• We apply a novel Mixture of Experts (MoE) extension pipeline to pretrained BERT models, where every multi-layer perceptron section is enlarged and copied into multiple distinct experts.