Encoder-decoder networks are popular for modeling sequences probabilisticallyin many applications. These models use the power of the Long Short-Term Memory(LSTM) architecture to capture the full dependence among variables, unlikeearlier models like CRFs that typically assumed conditional independence amongnon-adjacent variables. However in practice encoder-decoder models exhibit abias towards short sequences that surprisingly gets worse with increasing beamsize. In this paper we show that such phenomenon is due to a discrepancy betweenthe full sequence margin and the per-element margin enforced by the locallyconditioned training objective of a encoder-decoder model. The discrepancy moreadversely impacts long sequences, explaining the bias towards predicting shortsequences. For the case where the predicted sequences come from a closed set, we showthat a globally conditioned model alleviates the above problems ofencoder-decoder models. From a practical point of view, our proposed model alsoeliminates the need for a beam-search during inference, which reduces to anefficient dot-product based search in a vector-space.
展开▼
