Efficient GPT Model Pre-training using Tensor Train Matrix Representation

Understand

Large-scale transformer models have shown remarkable performance in language modelling tasks.

• However, such models feature billions of parameters, leading to difficulties in their deployment and prohibitive training costs from scratch.
• To reduce the number of the parameters in the GPT-2 architecture, we replace the matrices of fully-connected layers with the corresponding Tensor Train Matrix~(TTM) structure.
• Finally, we customize forward and backward operations through the TTM-based layer for simplicity and the stableness of further training.