Deepening Hidden Representations from Pre-trained Language Models

Understand

Transformer-based pre-trained language models have proven to be effective for learning contextualized language representation.

• However, current approaches only take advantage of the output of the encoder's final layer when fine-tuning the downstream tasks.
• We argue that only taking single layer's output restricts the power of pre-trained representation.
• Thus we deepen the representation learned by the model by fusing the hidden representation in terms of an explicit HIdden Representation Extractor (HIRE), which automatically absorbs the complementary representation with respect to the output from the final layer.