Using the Output Embedding to Improve Language Models

Understand

We study the topmost weight matrix of neural network language models.

• We show that this matrix constitutes a valid word embedding.
• When training language models, we recommend tying the input embedding and this output embedding.
• We analyze the resulting update rules and show that the tied embedding evolves in a more similar way to the output embedding than to the input embedding in the untied model.