2019

Distributionally Robust Language Modeling

Oren, Yonatan, Sagawa, Shiori, Hashimoto, Tatsunori B. et al.

Understand

Language models are generally trained on data spanning a wide range of topics (e.g., news, reviews, fiction), but they might be applied to an a priori unknown target distribution (e.g., restaurant reviews).

In this paper, we first show that training on text outside the test distribution can degrade test performance when using standard maximum likelihood (MLE) training.
To remedy this without the knowledge of the test distribution, we propose an approach which trains a model that performs well over a wide range of potential test distributions.
In particular, we derive a new distributionally robust optimization (DRO) procedure which minimizes the loss of the model over the worst-case mixture of topics with sufficient overlap with the training distribution.

Built on

Optimization of conditional value-at-risk
R. T. Rockafellar and S. Uryasev. 2000 · 2000
Earlier work this paper cites.
Improving predictive inference under covariate shift by weighting the log-likelihood function
H. Shimodaira. 2000 · 2000
Earlier work this paper cites.
Latent Dirichlet allocation
D. Blei, A. Ng, and M. I. Jordan. 2003 · 2003
Earlier work this paper cites.
Frustratingly easy domain adaptation
H. Daume III. 2007 · 2007
Earlier work this paper cites.
Domain adaptation with multiple sources
Y. Mansour, M. Mohri, and A. Rostamizadeh. 2009 · 2009
Earlier work this paper cites.
Dataset shift in machine learning
J. Quiñonero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. 2009 · 2009
Earlier work this paper cites.
A theory of learning from different domains
S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. 2010 · 2010
Earlier work this paper cites.
Domain adaptation with coupled subspaces
J. Blitzer, S. Kakade, and D. P. Foster. 2011 · 2011
Earlier work this paper cites.
Topic models for dynamic translation model adaptation
V. Eidelman, J. Boyd-Graber, and P. Resnik. 2012 · 2012
Earlier work this paper cites.
Discovering latent domains for multisource domain adaptation
J. Hoffman, B. Kulis, T. Darrell, and K. Saenko. 2012 · 2012
Earlier work this paper cites.
Robust solutions of optimization problems affected by uncertain probabilities
A. Ben-Tal, D. den Hertog, A. D. Waegenaere, B. Melenberg, and G. Rennen. 2013 · 2013
Earlier work this paper cites.
One billion word benchmark for measuring progress in statistical language modeling
Original
C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. 2013 · 2013
Earlier work this paper cites.
Reshaping visual datasets for domain adaptation
B. Gong, K. Grauman, and F. Sha. 2013 · 2013
Earlier work this paper cites.
Decoding with large-scale neural language models improves translation
A. Vaswani, Y. Zhao, V. Fossum, and D. Chiang. 2013 · 2013
Earlier work this paper cites.

Similar

Polylingual tree-based topic models for translation domain adaptation
Y. Hu, K. Zhai, V. Eidelman, and J. Boyd-Graber. 2014 · 2014
Cited alongside, same era.
Hierarchical multi-label conditional random fields for aspect-oriented opinion mining
D. Marcheggiani, O. Täckström, A. Esuli, and F. Sebastiani. 2014 · 2014
Cited alongside, same era.
Deep domain confusion: Maximizing for domain invariance
Original
E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell. 2014 · 2014
Cited alongside, same era.
Latent domains modeling for visual domain adaptation
C. Xiong, S. McCloskey, S. Hsieh, and J. J. Corso. 2014 · 2014
Cited alongside, same era.
Semi-supervised sequence learning
A. M. Dai and Q. V. Le. 2015 · 2015
Cited alongside, same era.
Unsupervised domain adaptation by backpropagation
Y. Ganin and V. Lempitsky. 2015 · 2015
Cited alongside, same era.
An improved non-monotonic transition system for dependency parsing
M. Honnibal and M. Johnson. 2015 · 2015
Cited alongside, same era.
A neural network approach to context-sensitive generation of conversational responses
A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015 · 2015
Cited alongside, same era.
Lightlda: Big topic models on modest compute clusters
J. Yuan, F. Gao, Q. Ho, W. Dai, J. Wei, X. Zheng, E. P. Xing, T. Liu, and W. Ma. 2015 · 2015
Cited alongside, same era.
Statistics of robust optimization: A generalized empirical likelihood approach
J. Duchi, P. Glynn, and H. Namkoong. 2016 · 2016
Cited alongside, same era.
Variance-based regularization with convex objectives
Original
J. Duchi and H. Namkoong. 2016 · 2016
Cited alongside, same era.
Abstractive text summarization using sequence-to-sequence rnns and beyond
Original
R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang, et al. 2016 · 2016
Cited alongside, same era.

Then

Learned in translation: Contextualized word vectors
B. McCann, J. Bradbury, C. Xiong, and R. Socher. 2017 · 2017
Later among the works it cites.
Variance regularization with convex objectives
H. Namkoong and J. Duchi. 2017 · 2017
Later among the works it cites.
Effective domain mixing for neural machine translation
R. Pryzant, D. Britz, and Q. V. Le. 2017 · 2017
Later among the works it cites.
Attention is all you need
Original
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017 · 2017
Later among the works it cites.
Yelp Dataset Challenge, Round 8
Yelp. 2017 · 2017
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2018 · 2018
Later among the works it cites.
Learning models with uniform performance via distributionally robust optimization
Original
J. Duchi and H. Namkoong. 2018 · 2018
Later among the works it cites.
Fairness without demographics in repeated loss minimization
T. B. Hashimoto, M. Srivastava, H. Namkoong, and P. Liang. 2018 · 2018
Later among the works it cites.
Algorithms and theory for multiple-source adaptation
J. Hoffman, M. Mohri, and N. Zhang. 2018 · 2018
Later among the works it cites.
Does distributionally robust supervised learning give robust classifiers?
W. Hu, G. Niu, I. Sato, and M. Sugiyama. 2018 · 2018
Later among the works it cites.
Deep contextualized word representations
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018 · 2018
Later among the works it cites.
Improving language understanding by generative pre-training
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018 · 2018
Later among the works it cites.
Certifiable distributional robustness with principled adversarial training
A. Sinha, H. Namkoong, and J. Duchi. 2018 · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…