2017

How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks

Jastrzebski, Stanisław, Leśniak, Damian, Czarnecki, Wojciech Marian

Understand

Maybe the single most important goal of representation learning is making subsequent learning faster.

Surprisingly, this fact is not well reflected in the way embeddings are evaluated.
In addition, recent practice in word embeddings points towards importance of learning specialized representations.
We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible.

Built on

Placing search in context: The concept revisited
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin · 2001
Earlier work this paper cites.
Domain adaptation for large-scale sentiment classification: A deep learning approach
Xavier Glorot, Antoine Bordes, and Yoshua Bengio · 2011
Earlier work this paper cites.
Representation learning: A review and new perspectives
Y. Bengio, A. Courville, and P. Vincent · 2013
Earlier work this paper cites.
Better word representations with recursive neural networks for morphology
Minh-Thang Luong, Richard Socher, and Christopher D. Manning · 2013
Earlier work this paper cites.
Recursive deep models for semantic compositionality over a sentiment treebank
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts · 2013
Earlier work this paper cites.
Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors
Marco Baroni and Georgiana et al. Dinu · 2014
Earlier work this paper cites.
Multimodal distributional semantics
Elia Bruni, Nam Khanh Tran, and Marco Baroni · 2014
Earlier work this paper cites.
Wordrep: A benchmark for research on learning word representations
Original
Bin Gao, Jiang Bian, and Tie-Yan Liu · 2014
Earlier work this paper cites.
Embedding word similarity with neural machine translation
Original
Felix Hill, Kyunghyun Cho, Sébastien Jean, Coline Devin, and Yoshua Bengio · 2014
Earlier work this paper cites.
Convolutional neural networks for sentence classification
Yoon Kim · 2014
Earlier work this paper cites.
Word embeddings through hellinger pca
Rémi Lebret and Ronan Collobert · 2014
Earlier work this paper cites.
Large-scale linear ranksvm
Ching-Pei Lee and Chih-Jen Lin · 2014
Earlier work this paper cites.

Similar

Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher Manning · 2014
Cited alongside, same era.
Learning word representations from scarce and noisy data with embedding subspaces
Ramón Fernández Astudillo, Silvio Amir, Wang Ling, Mário J. Silva, and Isabel Trancoso · 2015
Cited alongside, same era.
A Primer on Neural Network Models for Natural Language Processing, October 2015
Original
Yoav Goldberg · 2015
Cited alongside, same era.
Simlex-999: Evaluating semantic models with (genuine) similarity estimation
Felix Hill, Roi Reichart, and Anna Korhonen · 2015
Cited alongside, same era.
"the sum of its parts": Joint learning of word and phrase representations with autoencoders
Original
Rémi Lebret and Ronan Collobert · 2015
Cited alongside, same era.
Improving distributional similarity with lessons learned from word embeddings
Omer Levy, Yoav Goldberg, and Ido Dagan · 2015
Cited alongside, same era.
Two/too simple adaptations of word2vec for syntax problems
Wang Ling, Chris Dyer, Alan Black, and Isabel Trancoso · 2015
Cited alongside, same era.
Big data small data, in domain out-of domain, known word unknown word: The impact of word representation on sequence labelling tasks
Original
Lizhen Qu, Gabriela Ferraro, Liyuan Zhou, Weiwei Hou, Nathan Schneider, and Timothy Baldwin · 2015
Cited alongside, same era.
Evaluation methods for unsupervised word embeddings
Tobias Schnabel and Igor et al. Labutov · 2015
Cited alongside, same era.
Learning word representations by jointly modeling syntagmatic and paradigmatic relations
Fei Sun, Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng · 2015
Cited alongside, same era.
Character-level convolutional networks for text classification
Xiang Zhang, Junbo Zhao, and Yann LeCun · 2015
Cited alongside, same era.
How much do word embeddings encode about syntax?
Jacob Andreas and Dan Klein
Cited in the paper.

Then

A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification
Original
Ye Zhang and Byron Wallace · 2015
Later among the works it cites.
Linear algebraic structure of word senses, with applications to polysemy
Original
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski · 2016
Later among the works it cites.
Improving reliability of word similarity evaluation by redesigning annotation task and performance measure
Original
Oded Avraham and Yoav Goldberg · 2016
Later among the works it cites.
Intrinsic evaluation of word vectors fails to predict extrinsic performance
Billy Chiu, Anna Korhonen, and Sampo Pyysalo · 2016
Later among the works it cites.
Problems With Evaluation of Word Embeddings Using Word Similarity Tasks
M. Faruqui, Y. Tsvetkov, P. Rastogi, and C. Dyer · 2016
Later among the works it cites.
How to generate a good word embedding
Siwei Lai, Kang Liu, Shizhu He, and Jun Zhao · 2016
Later among the works it cites.
Neural architectures for named entity recognition
Guillaume Lample, Miguel Ballesteros, Kazuya Kawakami, Sandeep Subramanian, and Chris Dyer · 2016
Later among the works it cites.
Evaluating word embeddings using a representative suite of practical tasks
Neha Nayak, Gabor Angeli, and Christopher D. Manning · 2016
Later among the works it cites.
Sparsifying word representations for deep unordered sentence modeling
Prasanna Sattigeri and Jayaraman J Thiagarajan · 2016
Later among the works it cites.
Expanding subjective lexicons for social media mining with embedding subspaces
Original
Silvio Amir, Ramón Fernández Astudillo, Wang Ling, Paula C. Carvalho, and Mário J. Silva · 2017
Closest in time.
Word embedding calculus in meaningful ultradense subspaces
Sascha Rothe and Hinrich Schütze · 2083
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…