Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet

Understand

Recent years have seen a proliferation of attention mechanisms and the rise of Transformers in Natural Language Generation (NLG).

• Previously, state-of-the-art NLG architectures such as RNN and LSTM ran into vanishing gradient problems; as sentences grew larger, distance between positions remained linear, and sequential computation hindered parallelization since sentences were processed word by word.
• Transformers usher in a new era.
• In this paper, we explore three major Transformer-based models, namely GPT, BERT, and XLNet, that carry significant implications for the field.