2020

Language Model is All You Need: Natural Language Understanding as Question Answering

Namazifar, Mahdi, Papangelis, Alexandros, Tur, Gokhan et al.

Understand

Different flavors of transfer learning have shown tremendous impact in advancing research and applications of machine learning.

In this work we study the use of a specific family of transfer learning, where the target domain is mapped to the source domain.
Specifically we map Natural Language Understanding (NLU) problems to QuestionAnswering (QA) problems and we show that in low data regimes this approach offers significant improvements compared to other approaches to NLU.
Moreover we show that these gains could be increased through sequential transfer learning across NLU problems from different domains.

Built on

“The ATIS spoken language systems pilot corpus,”
Charles T. Hemphill, John J. Godfrey, and George R. Doddington, · 1990
Earlier work this paper cites.
“What is left to be understood in atis?,”
G. Tur, D. Hakkani-Tür, and L. Heck, · 2010
Earlier work this paper cites.
“Distributed representations of words and phrases and their compositionality,”
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean, · 2013
Earlier work this paper cites.
“Glove: Global vectors for word representation,”
Jeffrey Pennington, Richard Socher, and Christopher D. Manning, · 2014
Earlier work this paper cites.
“Towards zero-shot frame semantic parsing for domain scaling,”
Original
Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck, · 2017
Earlier work this paper cites.
“Attention is all you need,”
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin, · 2017
Earlier work this paper cites.
“A survey on deep transfer learning,”
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu, · 2018
Earlier work this paper cites.
“The natural language decathlon: Multitask learning as question answering,”
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher, · 2018
Earlier work this paper cites.
“BERT: Pre-training of deep bidirectional transformers for language understanding,”
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, · 2019
Earlier work this paper cites.

Similar

“Exploring the limits of transfer learning with a unified text-to-text transformer,”
Original
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and P. Liu, · 2019
Cited alongside, same era.
“BERT for joint intent classification and slot filling,”
Qian Chen, Zhu Zhuo, and Wen Wang, · 2019
Cited alongside, same era.
“Dialog state tracking: A neural reading comprehension approach,”
Original
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Hakkani-Tur, · 2019
Cited alongside, same era.
“Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering,”
Original
Li Zhou and Kevin Small, · 2019
Cited alongside, same era.
“Robust zero-shot cross-domain slot filling with example values,” 2019
Darsh J Shah, Raghav Gupta, Amir A Fayazi, and Dilek Hakkani-Tur, · 2019
Cited alongside, same era.
“Huggingface’s transformers: State-of-the-art natural language processing,”
Original
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush, · 2019
Cited alongside, same era.
“Joint slot filling and intent detection via capsule neural networks,”
Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip S Yu, · 2019
Cited alongside, same era.
“A comprehensive survey on transfer learning,” 2020
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He, · 2020
Cited alongside, same era.
“Know what you don’t know: Unanswerable questions for SQuAD,”
Pranav Rajpurkar, Robin Jia, and Percy Liang,
Cited in the paper.

Then

“A survey on transfer learning in natural language processing,”
Original
Zaid Alyafeai, Maged S. Al-shaibani, and I. Ahmad, · 2020
Closest in time.
“Learning to classify intents and slot labels given a handful of examples,”
Jason Krone, Yi Zhang, and Mona Diab, · 2020
Closest in time.
“Span-convert: Few-shot span extraction for dialog with pretrained conversational representations,” 2020
Sam Coope, Tyler Farghly, Daniela Gerz, Ivan Vulić, and Matthew Henderson, · 2020
Closest in time.
“From machine reading comprehension to dialogue state tracking: Bridging the gap,”
Original
Shuyang Gao, Sanchit Agarwal, Tagyoung Chung, Di Jin, and Dilek Hakkani-Tur, · 2020
Closest in time.
“Meta learning for few-shot joint intent detection and slot-filling,”
Hemanthage S Bhathiya and Uthayasanker Thayasivam, · 2020
Closest in time.
“Language models as few-shot learner for task-oriented dialogue systems,”
Original
Andrea Madotto, · 2020
Closest in time.
“Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” 2020
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf, · 2020
Closest in time.
“Albert: A lite bert for self-supervised learning of language representations,”
Original
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…