Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models

Understand

We present Branch-Train-Merge (BTM), a communication-efficient algorithm for embarrassingly parallel training of large language models (LLMs).

• We show it is possible to independently train subparts of a new class of LLMs on different subsets of the data, eliminating the massive multi-node synchronization currently required to train LLMs.
• BTM learns a set of independent expert LMs (ELMs), each specialized to a different textual domain, such as scientific or legal text.
• These ELMs can be added and removed to update data coverage, ensembled to generalize to new domains, or averaged to collapse back to a single LM for efficient inference.