Understand
We present Submatrix-wise Vector Embedding Learner (Swivel), a method for generating low-dimensional feature embeddings from a feature co-occurrence matrix.
- Swivel performs approximate factorization of the point-wise mutual information matrix via stochastic gradient descent.
- It uses a piecewise loss with special handling for unobserved co-occurrences, and thus makes use of all the information in the matrix.
- While this requires computation proportional to the size of the entire matrix, we make use of vectorized multiplication to process thousands of rows and columns at once to compute millions of predicted values.
Built on
Placing search in context: The concept revisited
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin · 2001
Earlier work this paper cites.
Collaborative filtering for implicit feedback datasets
Yifan Hu, Yehuda Koren, and Chris Volinsky · 2008
Earlier work this paper cites.
Using wiktionary for computing semantic relatedness
Torsten Zesch, Christof Müller, and Iryna Gurevych · 2008
Earlier work this paper cites.
A study on similarity and relatedness using distributional and wordnet-based approaches
Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa · 2009
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
John Duchi, Elad Hazan, and Yoram Singer · 2011
Earlier work this paper cites.
Similar
A word at a time: computing word relatedness using temporal semantic analysis
Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch · 2011
Cited alongside, same era.
Hogwild: A lock-free approach to parallelizing stochastic gradient descent
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu · 2011
Cited alongside, same era.
Distributional semantics in technicolor
Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran · 2012
Cited alongside, same era.
Large scale distributed deep networks
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al · 2012
Cited alongside, same era.
Better word representations with recursive neural networks for morphology
Minh-Thang Luong, Richard Socher, and Christopher D Manning · 2013
Cited alongside, same era.
Efficient estimation of word representations in vector space
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean
Cited in the paper.
Linguistic regularities in continuous space word representations
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig
Cited in the paper.
Then
Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors
Marco Baroni, Georgiana Dinu, and Germán Kruszewski · 2014
Later among the works it cites.
Simlex-999: Evaluating semantic models with (genuine) similarity estimation
Felix Hill, Roi Reichart, and Anna Korhonen · 2014
Later among the works it cites.
Neural word embedding as implicit matrix factorization
Omer Levy and Yoav Goldberg · 2014
Later among the works it cites.
Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher D Manning · 2014
Later among the works it cites.
Improving distributional similarity with lessons learned from word embeddings
Omer Levy, Yoav Goldberg, and Ido Dagan · 2015
Later among the works it cites.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…