2019

Calibration of Encoder Decoder Models for Neural Machine Translation

Kumar, Aviral, Sarawagi, Sunita

Understand

We study the calibration of several state of the art neural machine translation(NMT) systems built on attention-based encoder-decoder models.

For structured outputs like in NMT, calibration is important not just for reliable confidence with predictions, but also for proper functioning of beam-search inference.
We show that most modern NMT models are surprisingly miscalibrated even when conditioned on the true previous tokens.
Our investigation leads to two main reasons -- severe miscalibration of EOS (end of sequence marker) and suppression of attention uncertainty.

Built on

Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods
John C. Platt. 1999 · 1999
Earlier work this paper cites.
The Elements of Statistical Learning
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2001 · 2001
Earlier work this paper cites.
Transforming classifier scores into accurate multiclass probability estimates
Bianca Zadrozny and Charles Elkan. 2002 · 2002
Earlier work this paper cites.
Evaluating predictive uncertainty challenge
Joaquin Quiñonero Candela, Carl Edward Rasmussen, Fabian H. Sinz, Olivier Bousquet, and Bernhard Schölkopf. 2005 · 2005
Earlier work this paper cites.
Predicting good probabilities with supervised learning
Alexandru Niculescu-Mizil and Rich Caruana. 2005 · 2005
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Scheduled sampling for sequence prediction with recurrent neural networks
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015 · 2015
Earlier work this paper cites.
Calibrated structured prediction
Volodymyr Kuleshov and Percy S Liang. 2015 · 2015
Earlier work this paper cites.
Obtaining well calibrated probabilities using bayesian binning
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015 · 2015
Earlier work this paper cites.
Posterior calibration and exploratory analysis for natural language processing models
Khanh Nguyen and Brendan O’Connor. 2015 · 2015
Earlier work this paper cites.

Similar

Assessing calibration of prognostic risk scores
Cynthia S Crowson, Elizabeth J Atkinson, and Terry M Therneau. 2016 · 2016
Cited alongside, same era.
Improved neural machine translation with smt features
Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016 · 2016
Cited alongside, same era.
Sequence level training with recurrent neural networks
M Ranzato, S Chopra, M Auli, and W Zaremba. 2016 · 2016
Cited alongside, same era.
Length bias in encoder decoder models and a case for global conditioning
Pavel Sountsov and Sunita Sarawagi. 2016 · 2016
Cited alongside, same era.
Sequence-to-sequence learning as beam-search optimization
Sam Wiseman and Alexander M. Rush. 2016 · 2016
Cited alongside, same era.
On calibration of modern neural networks
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017 · 2017
Cited alongside, same era.
A baseline for detecting misclassified and out-of-distribution examples in neural networks
Dan Hendrycks and Kevin Gimpel. 2017 · 2017
Cited alongside, same era.
Six challenges for neural machine translation
Philipp Koehn and Rebecca Knowles. 2017 · 2017
Cited alongside, same era.
Simple and scalable predictive uncertainty estimation using deep ensembles
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017 · 2017
Cited alongside, same era.
Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation
Yilin Yang, Liang Huang, and Mingbo Ma. 2018a
Cited in the paper.

Then

Multiplicative normalizing flows for variational Bayesian neural networks
Christos Louizos and Max Welling. 2017 · 2017
Later among the works it cites.
Neural machine translation (seq2seq) tutorial
Minh-Thang Luong, Eugene Brevdo, and Rui Zhao. 2017 · 2017
Later among the works it cites.
Regularizing neural networks by penalizing confident output distributions
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017 · 2017
Later among the works it cites.
On fairness and calibration
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon M. Kleinberg, and Kilian Q. Weinberger. 2017 · 2017
Later among the works it cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Later among the works it cites.
Fast and scalable bayesian deep learning by weight-perturbation in adam
Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. 2018 · 2018
Later among the works it cites.
Accurate uncertainties for deep learning using calibrated regression
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. 2018 · 2018
Later among the works it cites.
Trainable calibration measures from kernel mean embeddings
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. 2018 · 2018
Later among the works it cites.
Analyzing uncertainty in neural machine translation
Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018 · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…