2019

Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System

Yang, Ze, Shou, Linjun, Gong, Ming et al.

Understand

Deep pre-training and fine-tuning models (such as BERT and OpenAI GPT) have demonstrated excellent results in question answering areas.

However, due to the sheer amount of model parameters, the inference speed of these models is very slow.
How to apply these complex models to real business scenarios becomes a challenging but practical problem.
Previous model compression methods usually suffer from information loss during the model compression procedure, leading to inferior models compared with the original one.

Built on

Distilling Word Embeddings: An Encoding Approach. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016 . 1977–1980
Lili Mou, Ran Jia, Yan Xu, Ge Li, Lu Zhang, and Zhi Jin. 2016 · 1980
Earlier work this paper cites.
Optimal Brain Damage. In Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989] . 598–605
Yann LeCun, John S. Denker, and Sara A. Solla. 1989 · 1989
Earlier work this paper cites.
Efficient and accurate approximations of nonlinear convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015 . 1984–1992
Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, and Jian Sun. 2015 · 1992
Earlier work this paper cites.
Second order derivatives for network pruning: Optimal Brain Surgeon
Babak Hassibi and David G. Stork. 1993 · 1993
Earlier work this paper cites.
A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning . ACM, 160–167
Ronan Collobert and Jason Weston. 2008 · 2008
Earlier work this paper cites.
Ontology-based interpretation of natural language
Philipp Cimiano, Christina Unger, and John McCrae. 2014 · 2014
Earlier work this paper cites.
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada . 1269–1277
Emily L. Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. 2014 · 2014
Earlier work this paper cites.
Inferring user demographics and social strategies in mobile social networks. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining . ACM, 15–24
Yuxiao Dong, Yang Yang, Jie Tang, Yang Yang, and Nitesh V Chawla. 2014 · 2014
Earlier work this paper cites.
Speeding up convolutional neural networks with low rank expansions
Original
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. 2014 · 2014
Earlier work this paper cites.
Distilling the Knowledge in a Neural Network
Geoffrey E Hinton, Oriol Vinyals, and Jeffrey Dean. 2015 · 2015
Earlier work this paper cites.
Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
Original
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David J. Crandall, and Dhruv Batra. 2015 · 2015
Earlier work this paper cites.

Similar

Rotating your face using multi-task deep neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 676–684
Junho Yim, Heechul Jung, ByungIn Yoo, Changkyu Choi, Dusik Park, and Junmo Kim. 2015 · 2015
Cited alongside, same era.
Sequence-Level Knowledge Distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016 . 1317–1327
Yoon Kim and Alexander M. Rush. 2016 · 2016
Cited alongside, same era.
Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016 . 1744–1753
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016 · 2016
Cited alongside, same era.
Channel Pruning for Accelerating Very Deep Neural Networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 . 1398–1406
Yihui He, Xiangyu Zhang, and Jian Sun. 2017 · 2017
Cited alongside, same era.
Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings
Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian J. Goodfellow, and Kunal Talwar. 2017 · 2017
Cited alongside, same era.
Multi-Task Learning with Labeled and Unlabeled Tasks
Anastasia Pentina and Christoph H Lampert. 2017 · 2017
Cited alongside, same era.
Learning from Multiple Teacher Networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017 . 1285–1294
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. 2017 · 2017
Cited alongside, same era.
Transfer and Multi-Task Learning for Noun-Noun Compound Interpretation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018 . 1488–1498
Murhaf Fares, Stephan Oepen, and Erik Velldal. 2018 · 2018
Cited alongside, same era.
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Original
Jonathan Frankle and Michael Carbin. 2018 · 2018
Cited alongside, same era.
Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers . 328–339
Jeremy Howard and Sebastian Ruder. 2018 · 2018
Cited alongside, same era.

Then

Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers) . 2227–2237
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 · 2018
Later among the works it cites.
Model compression via distillation and quantization
Original
Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018 · 2018
Later among the works it cites.
Improving Language Understanding by Generative Pre-Training
Alec Radford. 2018 · 2018
Later among the works it cites.
BAM! Born-Again Multi-Task Networks for Natural Language Understanding. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers . 5931–5937
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. 2019 · 2019
Closest in time.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Closest in time.
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
Original
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a · 2019
Closest in time.
Multi-Task Deep Neural Networks for Natural Language Understanding. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers . 4487–4496
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b · 2019
Closest in time.
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 · 2019
Closest in time.
Multi-teacher Knowledge Distillation for Compressed Video Action Recognition on Deep Neural Networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019 . 2202–2206
Meng-Chieh Wu, Ching-Te Chiu, and Kun-Hsuan Wu. 2019 · 2019
Closest in time.
Reducing BERT Pre-Training Time from 3 Days to 76 Minutes
Original
Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. 2019 · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…