Fetching the paper…

Reading the bibliography…

2015

Towards a Better Understanding of Predict and Count Models

Keerthi, S. Sathiya, Schnabel, Tobias, Khanna, Rajiv

Original
PDF
Post on X

Understand

In a recent paper, Levy and Goldberg pointed out an interesting connection between prediction-based word embedding models and count models based on pointwise mutual information.

Under certain conditions, they showed that both models end up optimizing equivalent objective functions.
This paper explores this connection in more detail and lays out the factors leading to differences between these models.
We find that the most relevant differences from an optimization perspective are (i) predict models work in a low dimensional space where embedding vectors can interact heavily; (ii) since predict models have fewer parameters, they are less prone to overfitting.

Towards a Better Understanding of Predict and Count Models · Around

Built on

A. Mnih and G. Hinton. A scalable hierarchical distributed language model
2009
Earlier work this paper cites.
P. Turney and P. Pantel. From frequency to meaning: Vector space models of semantics
Original
2010
Earlier work this paper cites.
T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space
Original
2013
Earlier work this paper cites.
T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality
2013
Earlier work this paper cites.

Similar

A. Mnih and K. Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation
2013
Cited alongside, same era.
M. Baroni, G. Dinu and G. Kruszewski. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors
2014
Cited alongside, same era.
O. Levy and Y. Goldberg. Neural word embedding as implicit factorization
2014
Cited alongside, same era.
O. Levy and Y. Goldberg. Linguistic Regularities in Sparse and Explicit Word Representations
2014
Cited alongside, same era.

Then

J. Pennington, R. Socher, C. D. Manning. GloVe: Global vectors for word representation
2014
Later among the works it cites.
T. Shi and Z. Liu. Linking Glove with
2014
Later among the works it cites.
O. Levy, Y. Goldberg and I. Dagan. Improving distributional similarity with lessons learned from word embeddings
2015
Closest in time.
K. Stratos, M. Collins and D. Hsu. Model-based Word Embeddings from Decompositions of Count Matrices
2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…