Fetching the paper…

Reading the bibliography…

2021

Sequence Parallelism: Long Sequence Training from System Perspective

Li, Shenggui, Xue, Fuzhao, Baranwal, Chaitanya et al.

Original
PDF
Post on X

Understand

Transformer achieves promising results on various tasks.

However, self-attention suffers from quadratic memory requirements with respect to the sequence length.
Existing work focuses on reducing time and space complexity from an algorithm perspective.
In this work, we propose sequence parallelism, a memory-efficient parallelism method to help us break input sequence length limitation and train with longer sequences on GPUs efficiently.

Sequence Parallelism: Long Sequence Training from System Perspective · Around

Built on

“Bert: Pre-training of deep bidirectional transformers for language understanding”
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova · 2018
Earlier work this paper cites.
“Gpipe: Efficient training of giant neural networks using pipeline parallelism”
Original
Yanping Huang et al · 2018
Earlier work this paper cites.
“High resolution medical image analysis with spatial partitioning”
Original
Le Hou et al · 2019
Earlier work this paper cites.
“BERT with history answer embedding for conversational question answering”
Chen Qu et al · 2019
Earlier work this paper cites.
“Language Models are Unsupervised Multitask Learners”, 2019
Alec Radford et al · 2019
Earlier work this paper cites.
“Megatron-lm: Training multi-billion parameter language models using model parallelism”
Original
Mohammad Shoeybi et al · 2019
Earlier work this paper cites.
“Language models are few-shot learners”
Original
Tom Brown et al · 2020
Earlier work this paper cites.
“An image is worth 16x16 words: Transformers for image recognition at scale”
Original
Alexey Dosovitskiy et al · 2020
Earlier work this paper cites.
“ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing”
Original
Ahmed Elnaggar et al · 2020
Earlier work this paper cites.

Similar

“Gshard: Scaling giant models with conditional computation and automatic sharding”
Original
Dmitry Lepikhin et al · 2020
Cited alongside, same era.
“Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”
Colin Raffel et al · 2020
Cited alongside, same era.
“Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters”
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase and Yuxiong He · 2020
Cited alongside, same era.
“Linformer: Self-attention with linear complexity”
Original
Sinong Wang et al · 2020
Cited alongside, same era.
“An Embarrassingly Simple Model for Dialogue Relation Extraction”
Original
Fuzhao Xue, Aixin Sun, Hao Zhang and Eng Chng · 2020
Cited alongside, same era.

Then

“Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling”
Original
Wenxuan Zhou, Kevin Huang, Tengyu Ma and Jing Huang · 2020
Later among the works it cites.
“Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity”
Original
William Fedus, Barret Zoph and Noam Shazeer · 2021
Closest in time.
“Efficient Large-Scale Language Model Training on GPU Clusters”
Original
Deepak Narayanan et al · 2021
Closest in time.
“Recent advances in deep learning based dialogue systems: A systematic survey”
Original
Jinjie Ni et al · 2021
Closest in time.
“ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning”

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

“GDPNet: Refining Latent Multi-View Graph for Relation Extraction”

Fuzhao Xue, Aixin Sun, Hao Zhang and Eng Chng · 2020

Cited alongside, same era.

“Bert representations for video question answering”

Zekun Yang et al · 2020

Cited alongside, same era.

“Big bird: Transformers for longer sequences”

Manzil Zaheer et al · 2020

Cited alongside, same era.

“Span-based Localizing Network for Natural Language Video Localization”

Hao Zhang, Aixin Sun, Wei Jing and Joey Zhou · 2020

Cited alongside, same era.

Samyam Rajbhandari et al · 2021

Closest in time.

“ZeRO-Offload: Democratizing Billion-Scale Model Training”, 2021

Jie Ren et al · 2021

Closest in time.

“PSSM-Distil: Protein Secondary Structure Prediction (PSSP) on Low-Quality PSSM by Knowledge Distillation with Contrastive Learning”, 2021

Qin Wang et al · 2021

Closest in time.

“GSPMD: General and Scalable Parallelization for ML Computation Graphs”

Yuanzhong Xu et al · 2021

Closest in time.

“Natural Language Video Localization: A Revisit in Span-based Question Answering Framework”

Hao Zhang et al · 2021

Closest in time.