An Empirical Exploration of Skip Connections for Sequential Tagging

Understand

In this paper, we empirically explore the effects of various kinds of skip connections in stacked bidirectional LSTMs for sequential tagging.

• We investigate three kinds of skip connections connecting to LSTM cells: (a) skip connections to the gates, (b) skip connections to the internal states and (c) skip connections to the cell outputs.
• We present comprehensive experiments showing that skip connections to cell outputs outperform the remaining two.
• Furthermore, we observe that using gated identity functions as skip mappings works pretty well.