Fetching the paper…

Reading the bibliography…

2015

Distilling Word Embeddings: An Encoding Approach

Mou, Lili, Jia, Ran, Xu, Yan et al.

Original
PDF
Post on X

Understand

Distilling knowledge from a well-trained cumbersome network to a small one has recently become a new research topic, as lightweight neural networks with high performance are particularly in need in various resource-restricted systems.

This paper addresses the problem of distilling word embeddings for NLP tasks.
We propose an encoding approach to distill task-specific knowledge from a set of high-dimensional embeddings, which can reduce model complexity by a large margin as well as retain high accuracy, showing a good compromise between efficiency and performance.
Experiments in two tasks reveal the phenomenon that distilling knowledge from cumbersome embeddings is better than directly training neural networks with small embeddings.

Distilling Word Embeddings: An Encoding Approach · Around

Built on

A neural probabilistic language model
Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin · 2003
Earlier work this paper cites.
Hierarchical probabilistic neural network language model
F. Morin and Y. Bengio · 2005
Earlier work this paper cites.
Model compression
C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil · 2006
Earlier work this paper cites.
A unified architecture for natural language processing: Deep neural networks with multitask learning
R. Collobert and J. Weston · 2008
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean · 2013
Earlier work this paper cites.

Similar

Do deep nets really need to be deep?
J. Ba and R. Caruana · 2014
Cited alongside, same era.
Distilling the knowledge in a neural network
Original
G. Hinton, O. Vinyals, and J. Dean · 2014
Cited alongside, same era.
FitNets: Hints for thin deep nets
A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio · 2014
Cited alongside, same era.
Transferring knowledge from a RNN to a DNN
Original
W. Chan, N. R. Ke, and I. Lane · 2015
Cited alongside, same era.

Then

Compressing neural networks with the hashing trick
W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen · 2015
Closest in time.
Discriminative neural sentence modeling by tree-based convolution
L. Mou, H. Peng, G. Li, Y. Xu, L. Zhang, and Z. Jin · 2015
Closest in time.
Recurrent neural network training with dark knowledge transfer
Original
D. Wang, C. Liu, Z. Tang, Z. Zhang, and M. Zhao · 2015
Closest in time.
Classifying relations via long short term memory networks along shortest dependency paths
Y. Xu, L. Mou, G. Li, Y. Chen, H. Peng, and Z. Jin · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…