Learning K-way D-dimensional Discrete Code For Compact Embedding Representations

Understand

Embedding methods such as word embedding have become pillars for many applications containing discrete structures.

• Conventional embedding methods directly associate each symbol with a continuous embedding vector, which is equivalent to applying linear transformation based on "one-hot" encoding of the discrete symbols.
• Despite its simplicity, such approach yields number of parameters that grows linearly with the vocabulary size and can lead to overfitting.
• In this work we propose a much more compact K-way D-dimensional discrete encoding scheme to replace the "one-hot" encoding.