Boosting Distributed Training Performance of the Unpadded BERT Model

Understand

Pre-training models are an important tool in Natural Language Processing (NLP), while the BERT model is a classic pre-training model whose structure has been widely adopted by followers.

• It was even chosen as the reference model for the MLPerf training benchmark.
• The distributed training performance optimization of BERT models plays an important role in accelerating the solutions of most NLP tasks.
• BERT model often uses padding tensors as its inputs, leading to excessive redundant computations.