Tesseract: Parallelize the Tensor Parallelism Efficiently

Understand

Together with the improvements in state-of-the-art accuracies of various tasks, deep learning models are getting significantly larger.

• However, it is extremely difficult to implement these large models because limited GPU memory makes it impossible to fit large models into a single GPU or even a GPU server.
• Besides, it is highly necessary to reduce the training time for large models.
• Previous methods like Megatron-LM implemented a 1-Dimensional distributed method to use GPUs to speed up the training.