Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018 · 2018
Cited alongside, same era.
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L.; Yang, N.; Wang, W.; Wei, F.; Liu, X.; Wang, Y.; Gao, J.; Zhou, M.; and Hon, H.-W. 2019 · 2019
Cited alongside, same era.
Text Summarization with Pretrained Encoders
Liu, Y.; and Lapata, M. 2019 · 2019
Cited alongside, same era.
Pretrained Language Models for Dialogue Generation with Multiple Input Sources
Cao, Y.; Bi, W.; Fang, M.; and Tao, D. 2020 · 2020
Cited alongside, same era.
Dialogue Response Ranking Training with Large-Scale Human Feedback Data
Gao, X.; Zhang, Y.; Galley, M.; Brockett, C.; and Dolan, W. B. 2020 · 2020
Cited alongside, same era.
ConveRT: Efficient and Accurate Conversational Representations from Transformers
Henderson, M.; Casanueva, I.; Mrkšić, N.; Su, P.-H.; Wen, T.-H.; and Vulić, I. 2020 · 2020
Cited alongside, same era.
Reformer: The Efficient Transformer
Kitaev, N.; Kaiser, L.; and Levskaya, A. 2020 · 2020
Cited alongside, same era.
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2020 · 2020
Cited alongside, same era.
Sparse sinkhorn attention
Tay, Y.; Bahri, D.; Yang, L.; Metzler, D.; and Juan, D.-C. 2020 · 2020
Cited alongside, same era.
TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue
Wu, C.-S.; Hoi, S. C.; Socher, R.; and Xiong, C. 2020 · 2020
Cited alongside, same era.
Big Bird: Transformers for Longer Sequences
Zaheer, M.; Guruganesh, G.; Dubey, K. A.; Ainslie, J.; Alberti, C.; Ontañón, S.; Pham, P.; Ravula, A.; Wang, Q.; Yang, L.; and Ahmed, A. 2020 · 2020
Cited alongside, same era.
A Hierarchical Network for Abstractive Meeting Summarization with Cross-Domain Pretraining
Zhu, C.; Xu, R.; Zeng, M.; and Huang, X. 2020 · 2020
Cited alongside, same era.