2020

Distilling Knowledge for Fast Retrieval-based Chat-bots

Tahami, Amir Vakili, Ghajar, Kamyar, Shakery, Azadeh

Understand

Response retrieval is a subset of neural ranking in which a model selects a suitable response from a set of candidates given a conversation history.

Retrieval-based chat-bots are typically employed in information seeking conversational systems such as customer support agents.
In order to make pairwise comparisons between a conversation history and a candidate response, two approaches are common: cross-encoders performing full self-attention over the pair and bi-encoders encoding the pair separately.
The former gives better prediction quality but is too slow for practical use.

Built on

Do deep nets really need to be deep?. In Advances in neural information processing systems
Jimmy Ba and Rich Caruana. 2014 · 2014
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015 · 2015
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings
Diederik P. Kingma and Jimmy Ba. 2015 · 2015
Earlier work this paper cites.
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015 · 2015
Earlier work this paper cites.
Sequence-Level Knowledge Distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
Yoon Kim and Alexander M Rush. 2016 · 2016
Earlier work this paper cites.
AliMe Assist : An Intelligent Assistant for Creating an Innovative E-commerce Experience. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017
Feng-Lin Li, Minghui Qiu, Haiqing Chen, Xiongwei Wang, Xing Gao, Jun Huang, Juwei Ren, Zhongzhou Zhao, Weipeng Zhao, Lei Wang, Guwei Jin, and Wei Chu. 2017 · 2017
Earlier work this paper cites.
Automatic Differentiation in PyTorch. In NIPS Autodiff Workshop
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017 · 2017
Earlier work this paper cites.

Similar

Attention is all you need. In Advances in neural information processing systems
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
A Compare-Aggregate Model for Matching Text Sequences. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings
Shuohang Wang and Jing Jiang. 2017 · 2017
Cited alongside, same era.
Training Millions of Personalized Dialogue Agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018
Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018 · 2018
Cited alongside, same era.
From Eliza to XiaoIce: challenges and opportunities with social chatbots
Heung-Yeung Shum, Xiao-dong He, and Di Li. 2018 · 2018
Cited alongside, same era.
On-device neural language model based word prediction. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations
Seunghak Yu, Nilesh Kulkarni, Haejun Lee, and Jihie Kim. 2018 · 2018
Cited alongside, same era.
DSTC7 Task 1: Noetic End-to-End Response Selection. In 7th Edition of the Dialog System Technology Challenges at AAAI 2019
Lazaros Polymenakos Chulaka Gunasekara, Jonathan K. Kummerfeld and Walter S. Lasecki. 2019 · 2019
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.

Then

ConveRT: Efficient and Accurate Conversational Representations from Transformers
Original
Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Ivan Vulić, et al · 2019
Later among the works it cites.
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Original
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019 · 2019
Later among the works it cites.
Distilling task-specific knowledge from BERT into simple neural networks
Original
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019 · 2019
Later among the works it cites.
Multi-Representation Fusion Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining
Chongyang Tao, Wei Wu, Can Xu, Wenpeng Hu, Dongyan Zhao, and Rui Yan. 2019 · 2019
Later among the works it cites.
Poly-encoders: architectures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Conference on Learning Representations, ICLR 2020
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020 · 2020
Closest in time.
Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking. In European Conference on Information Retrieval
Gustavo Penha and Claudia Hauff. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…