2022

Improving Out-of-Distribution Robustness via Selective Augmentation

Yao, Huaxiu, Wang, Yu, Li, Sai et al.

Understand

Machine learning algorithms typically assume that training and test examples are drawn from the same distribution.

However, distribution shift is a common problem in real-world applications and can cause models to perform dramatically worse at test time.
In this paper, we specifically consider the problems of subpopulation shifts (e.g., imbalanced data) and domain shifts.
While prior works often seek to explicitly regularize internal representations or predictors of the model to be domain invariant, we instead aim to learn invariant predictors without restricting the model's internal representations or predictors.

Built on

An introduction to multivariate statistical analysis
Anderson, T. W · 1962
Earlier work this paper cites.
Domain generalization with optimal transport and metric learning
Original
Zhou, F., Jiang, Z., Shui, C., Wang, B., and Chaib-draa, B · 2007
Earlier work this paper cites.
A theory of learning from different domains
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W · 2010
Earlier work this paper cites.
The Caltech-UCSD Birds-200-2011 Dataset
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S · 2011
Earlier work this paper cites.
Domain generalization via invariant feature representation
Muandet, K., Balduzzi, D., and Schölkopf, B · 2013
Earlier work this paper cites.
Deep domain confusion: Maximizing for domain invariance
Original
Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., and Darrell, T · 2014
Earlier work this paper cites.
Unsupervised domain adaptation by backpropagation
Ganin, Y. and Lempitsky, V · 2015
Earlier work this paper cites.
Deep learning face attributes in the wild
Liu, Z., Luo, P., Wang, X., and Tang, X · 2015
Earlier work this paper cites.
Learning transferable features with deep adaptation networks
Long, M., Cao, Y., Wang, J., and Jordan, M · 2015
Earlier work this paper cites.
Mathematical Methods of Statistics (PMS-9), Volume 9
Cramér, H · 2016
Earlier work this paper cites.
Domain-adversarial training of neural networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Original
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., Bernstein, M., and Fei-Fei, L · 2016
Earlier work this paper cites.
Deep coral: Correlation alignment for deep domain adaptation
Sun, B. and Saenko, K · 2016
Earlier work this paper cites.
Densely connected convolutional networks
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q · 2017
Earlier work this paper cites.
Places: A 10 million image database for scene recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A · 2017
Earlier work this paper cites.
From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge
Bandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Bejnordi, B. E., Lee, B., Paeng, K., Zhong, A., et al · 2018
Earlier work this paper cites.
Functional map of the world
Christie, G., Fendley, N., Wilson, J., and Mukherjee, R · 2018
Earlier work this paper cites.
Domain generalization with adversarial feature learning
Li, H., Pan, S. J., Wang, S., and Kot, A. C · 2018
Earlier work this paper cites.
Generalizing to unseen domains via adversarial data augmentation
Original
Volpi, R., Namkoong, H., Sener, O., Duchi, J., Murino, V., and Savarese, S · 2018
Earlier work this paper cites.
mixup: Beyond empirical risk minimization
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D · 2018
Earlier work this paper cites.

Similar

Generalizing to unseen domains via distribution matching
Original
Albuquerque, I., Monteiro, J., Darvishi, M., Falk, T. H., and Mitliagkas, I · 2019
Cited alongside, same era.
Invariant risk minimization
Original
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D · 2019
Cited alongside, same era.
Nuanced metrics for measuring unintended bias with real data for text classification
Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasserman, L · 2019
Cited alongside, same era.
Learning imbalanced datasets with label-distribution-aware margin loss
Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T · 2019
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2019
Cited alongside, same era.
Meta dropout: Learning to perturb latent features for generalization
Lee, H. B., Nam, T., Yang, E., and Hwang, S. J · 2019
Cited alongside, same era.
The generalization error of max-margin linear classifiers: High-dimensional asymptotics in the overparametrized regime
Original
Montanari, A., Ruan, F., Sohn, Y., and Yan, J · 2019
Cited alongside, same era.
Justifying recommendations using distantly-labeled reviews and fine-grained aspects
Ni, J., Li, J., and McAuley, J · 2019
Cited alongside, same era.
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter
Original
Sanh, V., Debut, L., Chaumond, J., and Wolf, T · 2019
Cited alongside, same era.
Rxrx1: An image set for cellular morphological variation across many experimental batches
Taylor, J., Earnshaw, B., Mabey, B., Victors, M., and Yosinski, J · 2019
Cited alongside, same era.
High dimensional linear discriminant analysis: optimality, adaptive algorithm and missing data
Tony Cai, T. and Zhang, L · 2019
Cited alongside, same era.
Manifold mixup: Better representations by interpolating hidden states
Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., and Bengio, Y · 2019
Cited alongside, same era.

Then

Adversarial domain adaptation with domain mixup
Xu, M., Zhang, J., Ni, B., Li, T., Wang, C., Tian, Q., and Zhang, W · 2020
Later among the works it cites.
Improve unsupervised domain adaptation with mixup training
Original
Yan, S., Song, H., Li, N., Zou, L., and Ren, L · 2020
Later among the works it cites.
Maximum-entropy adversarial data augmentation for improved generalization and robustness
Original
Zhao, L., Liu, T., Peng, X., and Metaxas, D · 2020
Later among the works it cites.
Invariance principle meets information bottleneck for out-of-distribution generalization
Ahuja, K., Caballero, E., Zhang, D., Bengio, Y., Mitliagkas, I., and Rish, I · 2021
Later among the works it cites.
A convex optimization approach to high-dimensional sparse quadratic discriminant analysis
Cai, T. T. and Zhang, L · 2021
Later among the works it cites.
Fair mixup: Fairness via interpolation
Chuang, C.-Y. and Mroueh, Y · 2021
Later among the works it cites.
Environment inference for invariant learning
Creager, E., Jacobsen, J.-H., and Zemel, R · 2021
Later among the works it cites.
Model patching: Closing the subgroup performance gap with data augmentation
Goel, K., Gu, A., Li, Y., and Ré, C · 2021
Later among the works it cites.
Out-of-distribution prediction with invariant risk minimization: The limitation and an effective fix
Original
Guo, R., Zhang, P., Liu, H., and Kiciman, E · 2021
Later among the works it cites.
On invariance penalties for risk minimization
Original
Khezeli, K., Blaas, A., Soboczenski, F., Chia, N., and Kalantari, J · 2021
Later among the works it cites.
Wilds: A benchmark of in-the-wild distribution shifts
Koh, P. W., Sagawa, S., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., Lee, T., et al · 2021
Later among the works it cites.
Out-of-distribution generalization via risk extrapolation (rex)
Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Le Priol, R., and Courville, A · 2021
Later among the works it cites.
Metadataset: A dataset of datasets for evaluating distribution shifts and training conflicts
Liang, W. and Zou, J · 2021
Later among the works it cites.
The risks of invariant risk minimization
Rosenfeld, E., Ravikumar, P., and Risteski, A · 2021
Later among the works it cites.
Gradient matching for domain generalization
Original
Shi, Y., Seely, J., Torr, P. H., Siddharth, N., Hannun, A., Usunier, N., and Synnaeve, G · 2021
Later among the works it cites.
Open domain generalization with domain-augmented meta-learning
Shu, Y., Cao, Z., Wang, C., Wang, J., and Long, M · 2021
Later among the works it cites.
Meta-learning with fewer tasks through task interpolation
Yao, H., Zhang, L., and Finn, C · 2021
Later among the works it cites.
Correct-n-contrast: A contrastive approach for improving robustness to spurious correlations
Zhang, M., Sohoni, N. S., Zhang, H. R., Finn, C., and Ré, C · 2021
Later among the works it cites.
Examining and combating spurious features under distribution shift
Zhou, C., Ma, X., Michel, P., and Neubig, G · 2021
Later among the works it cites.
Diversify and disambiguate: Learning from underspecified data
Original
Lee, Y., Yao, H., and Finn, C · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…