2021

Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix

Guo, Ruocheng, Zhang, Pengchuan, Liu, Hao et al.

Understand

This work considers the out-of-distribution (OOD) prediction problem where (1)~the training data are from multiple domains and (2)~the test domain is unseen in the training.

DNNs fail in OOD prediction because they are prone to pick up spurious correlations.
Recently, Invariant Risk Minimization (IRM) is proposed to address this issue.
Its effectiveness has been demonstrated in the colored MNIST experiment.

Built on

Principles of risk minimization for learning theory
Vladimir Vapnik · 1992
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.
Robust optimization
Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski · 2009
Earlier work this paper cites.
A kernel two-sample test
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola · 2012
Earlier work this paper cites.
Optimal kernel choice for large-scale two-sample tests
Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur · 2012
Earlier work this paper cites.
Domain adaptation under target and conditional shift
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang · 2013
Earlier work this paper cites.
Unsupervised domain adaptation by backpropagation
Yaroslav Ganin and Victor Lempitsky · 2015
Earlier work this paper cites.
Learning transferable features with deep adaptation networks
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan · 2015
Earlier work this paper cites.
Domain adaptation with conditional transferable components
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf · 2016
Earlier work this paper cites.
Causal inference by using invariant prediction: identification and confidence intervals
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen · 2016
Earlier work this paper cites.

Similar

Estimating individual treatment effect: generalization bounds and algorithms
Uri Shalit, Fredrik D Johansson, and David Sontag · 2017
Cited alongside, same era.
Wasserstein auto-encoders
Original
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf · 2017
Cited alongside, same era.
Recognition in terra incognita
Sara Beery, Grant Van Horn, and Pietro Perona · 2018
Cited alongside, same era.
Deep domain generalization via conditional invariant adversarial networks
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao · 2018
Cited alongside, same era.
Conditional adversarial domain adaptation
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan · 2018
Cited alongside, same era.
Invariant risk minimization
Original
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz · 2019
Cited alongside, same era.
Invariant risk minimization games
Original
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar · 2020
Cited alongside, same era.
Empirical or invariant risk minimization? a sample complexity perspective
Original
Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney · 2020
Cited alongside, same era.
Environment inference for invariant learning
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel · 2020
Cited alongside, same era.
Underspecification presents challenges for credibility in modern machine learning
Original
Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al · 2020
Cited alongside, same era.

Then

Shortcut learning in deep neural networks
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann · 2020
Later among the works it cites.
In search of lost domain generalization
Original
Ishaan Gulrajani and David Lopez-Paz · 2020
Later among the works it cites.
Designing data augmentation for simulating interventions
Original
Maximilian Ilse, Jakub M Tomczak, and Patrick Forré · 2020
Later among the works it cites.
Domain extrapolation via regret minimization
Original
Wengong Jin, Regina Barzilay, and Tommi Jaakkola · 2020
Later among the works it cites.
Out-of-distribution generalization via risk extrapolation (rex)
Original
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Remi Le Priol, and Aaron Courville · 2020
Later among the works it cites.
The risks of invariant risk minimization
Original
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski · 2020
Later among the works it cites.
Novel human-object interaction detection via adversarial domain generalization
Original
Yuhang Song, Wenbo Li, Lei Zhang, Jianwei Yang, Emre Kiciman, Hamid Palangi, Jianfeng Gao, C-C Jay Kuo, and Pengchuan Zhang · 2020
Later among the works it cites.
A causal view on robustness of neural networks
Original
Cheng Zhang, Kun Zhang, and Yingzhen Li · 2020
Later among the works it cites.
Out-of-distribution generalization analysis via influence function
Original
Haotian Ye, Chuanlong Xie, Yue Liu, and Zhenguo Li · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…