Length bias in Encoder Decoder Models and a Case for Global Conditioning

Understand

Encoder-decoder networks are popular for modeling sequences probabilistically in many applications.

• These models use the power of the Long Short-Term Memory (LSTM) architecture to capture the full dependence among variables, unlike earlier models like CRFs that typically assumed conditional independence among non-adjacent variables.
• However in practice encoder-decoder models exhibit a bias towards short sequences that surprisingly gets worse with increasing beam size.
• In this paper we show that such phenomenon is due to a discrepancy between the full sequence margin and the per-element margin enforced by the locally conditioned training objective of a encoder-decoder model.