Embedding Words as Distributions with a Bayesian Skip-gram Model

Understand

We introduce a method for embedding words as probability densities in a low-dimensional space.

• Rather than assuming that a word embedding is fixed across the entire text collection, as in standard word embedding methods, in our Bayesian model we generate it from a word-specific prior density for each occurrence of a given word.
• Intuitively, for each word, the prior density encodes the distribution of its potential 'meanings'.
• These prior densities are conceptually similar to Gaussian embeddings.