Fetching the paper…

Reading the bibliography…

2024

Is Cosine-Similarity of Embeddings Really About Similarity?

Steck, Harald, Ekanadham, Chaitanya, Kallus, Nathan

Original
PDF
Post on X

Understand

Cosine-similarity is the cosine of the angle between two vectors, or equivalently the dot product between their normalizations.

A popular application is to quantify semantic similarity between high-dimensional objects by applying cosine-similarity to a learned low-dimensional feature embedding.
This can work better but sometimes also worse than the unnormalized dot-product between embedded vectors in practice.
To gain insight into this empirical observation, we study embeddings derived from regularized linear models, where closed-form solutions facilitate analytical insights.

Is Cosine-Similarity of Embeddings Really About Similarity? · Around

Built on

Dense passage retrieval for open-domain question answering, 2020
Original
V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih · 2004
Earlier work this paper cites.
ColBERT: Efficient and effective passage search via contextualized late interaction over BERT, 2020
Original
O. Khattab and M. Zaharia · 2004
Earlier work this paper cites.
Efficient estimation of word representations in vector space, 2013
T. Mikolov, K. Chen, G. Corrado, and J. Dean · 2013
Earlier work this paper cites.

Similar

Layer normalization, 2016
Original
J. L. Ba, J. R. Kiros, and G. E. Hinton · 2016
Cited alongside, same era.
Regularized singular value decomposition and application to recommender system, 2018
Original
S. Zheng, C. Ding, and F. Nie · 2018
Cited alongside, same era.
Autoencoders that don’t overfit towards the identity
H. Steck · 2020
Cited alongside, same era.

Then

Towards a better understanding of linear models for recommendation
R. Jin, D. Li, J. Gao, Z. Liu, L. Chen, and Y. Zhou · 2021
Later among the works it cites.
Problems with cosine as a measure of embedding similarity for high frequency words
K. Zhou, K. Ethayarajh, D. Card, and D. Jurafsky · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…