Generative Representational Instruction Tuning

Understand

All text-based language problems can be reduced to either generation or embedding.

• Current models only perform well at one or the other.
• We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions.
• Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks.