Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement

Understand

With the promising progress of deep neural networks, layer aggregation has been used to fuse information across layers in various fields, such as computer vision and machine translation.

• However, most of the previous methods combine layers in a static fashion in that their aggregation strategy is independent of specific hidden states.
• Inspired by recent progress on capsule networks, in this paper we propose to use routing-by-agreement strategies to aggregate layers dynamically.
• Specifically, the algorithm learns the probability of a part (individual layer representations) assigned to a whole (aggregated representations) in an iterative way and combines parts accordingly.