Efficient Contextual Representation Learning Without Softmax Layer

Understand

Contextual representation models have achieved great success in improving various downstream tasks.

• However, these language-model-based encoders are difficult to train due to the large parameter sizes and high computational complexity.
• By carefully examining the training procedure, we find that the softmax layer (the output layer) causes significant inefficiency due to the large vocabulary size.
• Therefore, we redesign the learning objective and propose an efficient framework for training contextual representation models.