2020

DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances

Gu, Xiaodong, Yoo, Kang Min, Ha, Jung-Woo

Understand

Recent advances in pre-trained language models have significantly improved neural response generation.

However, existing methods usually view the dialogue context as a linear sequence of tokens and learn to generate the next word through token-level self-attention.
Such token-level encoding hinders the exploration of discourse-level coherence among utterances.
This paper presents DialogBERT, a novel conversational response generation model that enhances previous PLM-based dialogue models.

Built on

The evolved transformer
Original
So, D. R.; Liang, C.; and Le, Q. V. 2019 · 1901
Earlier work this paper cites.
Transfertransfo: A transfer learning approach for neural network based conversational agents
Original
Wolf, T.; Sanh, V.; Chaumond, J.; and Delangue, C. 2019b · 1901
Earlier work this paper cites.
HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization
Original
Zhang, X.; Wei, F.; and Zhou, M. 2019 · 1905
Earlier work this paper cites.
Leveraging pre-trained checkpoints for sequence generation tasks
Original
Rothe, S.; Narayan, S.; and Severyn, A. 2019 · 1907
Earlier work this paper cites.
Ernie 2.0: A continual pre-training framework for language understanding
Original
Sun, Y.; Wang, S.; Li, Y.; Feng, S.; Tian, H.; Wu, H.; and Wang, H. 2019 · 1907
Earlier work this paper cites.
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019 · 1910
Earlier work this paper cites.
HuggingFace’s Transformers: State-of-the-art Natural Language Processing
Original
Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; and Brew, J. 2019a · 1910
Earlier work this paper cites.
ConveRT: Efficient and Accurate Conversational Representations from Transformers
Original
Henderson, M.; Casanueva, I.; Mrkšić, N.; Su, P.-H.; Vulić, I.; et al. 2019 · 1911
Earlier work this paper cites.
DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
Original
Zhang, Y.; Sun, S.; Galley, M.; Chen, Y.-C.; Brockett, C.; Gao, X.; Gao, J.; Liu, J.; and Dolan, B. 2019 · 1911
Earlier work this paper cites.
Towards a human-like open-domain chatbot
Original
Adiwardana, D.; Luong, M.-T.; So, D. R.; Hall, J.; Fiedel, N.; Thoppilan, R.; Yang, Z.; Kulshreshtha, A.; Nemade, G.; Lu, Y.; et al. 2020 · 2001
Earlier work this paper cites.
Deep Attentive Ranking Networks for Learning to Order Sentences
Original
Kumar, P.; Brahma, D.; Karnick, H.; and Rai, P. 2019 · 2001
Earlier work this paper cites.
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics
Doddington, G. 2002 · 2002
Earlier work this paper cites.
BLEU: a method for automatic evaluation of machine translation
Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002 · 2002
Earlier work this paper cites.

Similar

Recipes for building an open-domain chatbot
Original
Roller, S.; Dinan, E.; Goyal, N.; Ju, D.; Williamson, M.; Liu, Y.; Xu, J.; Ott, M.; Shuster, K.; Smith, E. M.; et al. 2020 · 2004
Cited alongside, same era.
Learning to rank: from pairwise approach to listwise approach
Cao, Z.; Qin, T.; Liu, T.-Y.; Tsai, M.-F.; and Li, H. 2007 · 2007
Cited alongside, same era.
Multi-label learning with millions of labels: Recommending advertiser bid phrases for web pages
Agrawal, R.; Gupta, A.; Prabhu, Y.; and Varma, M. 2013 · 2013
Cited alongside, same era.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
Original
Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014 · 2014
Cited alongside, same era.
Glove: Global vectors for word representation
Pennington, J.; Socher, R.; and Manning, C. D. 2014 · 2014
Cited alongside, same era.
Neural-based context representation learning for dialog act classification
Original
Ortega, D.; and Vu, N. T. 2017 · 2017
Cited alongside, same era.
Nsml: A machine learning platform that enables you to focus on your models
Original
Sung, N.; Kim, M.; Jo, H.; Yang, Y.; Kim, J.; Lausen, L.; Kim, Y.; Lee, G.; Kwak, D.; Ha, J.-W.; et al. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017 · 2017
Cited alongside, same era.
Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling
Original
Budzianowski, P.; Wen, T.-H.; Tseng, B.-H.; Casanueva, I.; Ultes, S.; Ramadan, O.; and Gašić, M. 2018 · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018 · 2018
Cited alongside, same era.
Decoupled Weight Decay Regularization
Loshchilov, I.; and Hutter, F. 2018 · 2018
Cited alongside, same era.
Deep Contextualized Word Representations
Peters, M.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018 · 2018
Cited alongside, same era.

Then

Improving variational encoder-decoders in dialogue generation
Shen, X.; Su, H.; Niu, S.; and Demberg, V. 2018 · 2018
Later among the works it cites.
Hello, It’s GPT-2-How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems
Budzianowski, P.; and Vulić, I. 2019 · 2019
Later among the works it cites.
DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder
Gu, X.; Cho, K.; Ha, J.-W.; and Kim, S. 2019 · 2019
Later among the works it cites.
G-Softmax: Improving Intraclass Compactness and Interclass Separability of Features
Luo, Y.; Wong, Y.; Kankanhalli, M.; and Zhao, Q. 2019 · 2019
Later among the works it cites.
Pretraining Methods for Dialog Context Representation Learning
Mehri, S.; Razumovskaia, E.; Zhao, T.; and Eskenazi, M. 2019 · 2019
Later among the works it cites.
Combining Unsupervised Pre-training and Annotator Rationales to Improve Low-shot Text Classification
Melamud, O.; Bornea, M.; and Barker, K. 2019 · 2019
Later among the works it cites.
VisualHyperTuner: Visual analytics for user-driven hyperparameter tuning of deep neural networks
Park, H.; Kim, J.; Kim, M.; Kim, J.-H.; Choo, J.; Ha, J.-W.; and Sung, N. 2019 · 2019
Later among the works it cites.
Language Models are Unsupervised Multitask Learners
Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019 · 2019
Later among the works it cites.
Xlnet: Generalized autoregressive pretraining for language understanding
Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V. 2019 · 2019
Later among the works it cites.
What you say and how you say it: Joint modeling of topics and discourse in microblog conversations
Zeng, J.; Li, J.; He, Y.; Gao, C.; Lyu, M. R.; and King, I. 2019 · 2019
Later among the works it cites.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K.; Luong, M.-T.; Le, Q. V.; and Manning, C. D. 2020 · 2020
Closest in time.
A Pre-Training Based Personalized Dialogue Generation Model with Persona-Sparse Data
Zheng, Y.; Zhang, R.; Huang, M.; and Mao, X. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…