2021

Generalized Out-of-Distribution Detection: A Survey

Yang, Jingkang, Zhou, Kaiyang, Li, Yixuan et al.

Understand

Out-of-distribution (OOD) detection is critical to ensuring the reliability and safety of machine learning systems.

For instance, in autonomous driving, we would like the driving system to issue an alert and hand over the control to humans when it detects unusual scenes or objects that it has never seen during training time and cannot make a safe decision.
The term, OOD detection, first emerged in 2017 and since then has received increasing attention from the research community, leading to a plethora of methods developed, ranging from classification-based to density-based to distance-based ones.
Meanwhile, several other problems, including anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and outlier detection (OD), are closely related to OOD detection in terms of motivation and methodology.

Built on

E. Parzen, “On estimation of a probability density function and mode,” The annals of mathematical statistics
1962
Earlier work this paper cites.
C. Chow, “On optimum recognition error and reject tradeoff,” IEEE Transactions on Information Theory
1970
Earlier work this paper cites.
J. Van Ryzin, “A histogram method of density estimation,” Communications in Statistics-Theory and Methods
1973
Earlier work this paper cites.
M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM
1981
Earlier work this paper cites.
R. A. Redner and H. F. Walker, “Mixture densities, maximum likelihood and the em algorithm,” SIAM review
1984
Earlier work this paper cites.
E. T. Jaynes, “Bayesian methods: General background,” 1986
1986
Earlier work this paper cites.
C. Peterson and E. Hartman, “Explorations of the mean field theory learning algorithm,” Neural Networks
1989
Earlier work this paper cites.
R. L. Smith, “Extreme value theory,” Handbook of applicable mathematics
1990
Earlier work this paper cites.
A. J. Izenman, “Review papers: Recent developments in nonparametric density estimation,” Journal of the American Statistical Association
1991
Earlier work this paper cites.
M. A. Kramer, “Nonlinear principal component analysis using autoassociative neural networks,” AIChE journal
1991
Earlier work this paper cites.
PhD thesis, California Institute of Technology, 1992
D. J. C. Mackay, Bayesian methods for adaptive models · 1992
Earlier work this paper cites.
D. Wettschereck, “A study of distance-based machine learning algorithms,” 1994
1994
Earlier work this paper cites.
C. Cortes and V. Vapnik, “Support-vector networks,” Machine learning
1995
Earlier work this paper cites.
M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al
1996
Earlier work this paper cites.
G. Danuser and M. Stricker, “Parametric model fitting: From inlier characterization to outlier detection,” TPAMI
1998
Earlier work this paper cites.
M. Desforges, P. Jacob, and J. Cooper, “Applications of probability density estimation to the detection of abnormal conditions in engineering,” Proceedings of the institution of mechanical engineers
1998
Earlier work this paper cites.
T. G. Dietterich, “Ensemble methods in machine learning,” in International workshop on multiple classifier systems
2000
Earlier work this paper cites.
R. De Maesschalck, D. Jouan-Rimbaud, and D. L. Massart, “The mahalanobis distance,” Chemometrics and intelligent laboratory systems
2000
Earlier work this paper cites.
E. Eskin, “Anomaly detection over noisy data using learned probability distributions,” in ICML
2000
Earlier work this paper cites.
M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “Lof: identifying density-based local outliers,” in SIGMOD
2000
Earlier work this paper cites.
C. C. Aggarwal and P. S. Yu, “Outlier detection for high dimensional data,” in ACM SIGMOD
2001
Earlier work this paper cites.
C. P. Diehl and J. B. Hampshire, “Real-time object classification and novelty detection for collaborative video surveillance,” in IJCNN
2002
Earlier work this paper cites.
G. Fumera and F. Roli, “Support vector machines with embedded reject option,” in International Workshop on Support Vector Machines
2002
Earlier work this paper cites.
D. M. J. Tax, “One-class classification: Concept learning in the absence of counter-examples.,” 2002
2002
Earlier work this paper cites.
M. Markou and S. Singh, “Novelty detection: a review—part 1: statistical approaches,” Signal processing
2003
Earlier work this paper cites.
M. Markou and S. Singh, “Novelty detection: a review—part 2: neural network based approaches,” Signal processing
2003
Earlier work this paper cites.
C. C. Noble and D. J. Cook, “Graph-based anomaly detection,” in SIGKDD
2003
Earlier work this paper cites.
V. Hodge and J. Austin, “A survey of outlier detection methodologies,” Artificial intelligence review
2004
Earlier work this paper cites.
H. Liu, S. Shah, and W. Jiang, “On-line outlier detection and data cleaning,” Computers & chemical engineering
2004
Earlier work this paper cites.
A. Loureiro, L. Torgo, and C. Soares, “Outlier detection using clustering methods: a data cleaning application,” in Proceedings of KDNet Symposium on Knowledge-based Systems
2004
Earlier work this paper cites.
V. Hautamaki, I. Karkkainen, and P. Franti, “Outlier detection using k-nearest neighbour graph,” in ICPR
2004
Earlier work this paper cites.
F. Muhlenbach, S. Lallich, and D. A. Zighed, “Identifying and handling mislabelled instances,” Journal of Intelligent Information Systems
2004
Earlier work this paper cites.
I. Ben-Gal, “Outlier detection,” in Data mining and knowledge discovery handbook
2005
Earlier work this paper cites.
I. Ben-Gal, “Outlier detection,” in Data mining and knowledge discovery handbook
2005
Earlier work this paper cites.
J. Van den Broeck, S. Argeseanu Cunningham, R. Eeckels, and K. Herbst, “Data cleaning: detecting, diagnosing, and editing data abnormalities,” PLoS medicine
2005
Earlier work this paper cites.
A. Hyvärinen and P. Dayan, “Estimation of non-normalized statistical models by score matching.,” 2005
2005
Earlier work this paper cites.
D. G. Altman and J. M. Bland, “Standard deviations and standard errors,” BMJ
2005
Earlier work this paper cites.
Y. LeCun and C. Cortes, “The mnist database of handwritten digits,” 2005
2005
Earlier work this paper cites.
N. Drummond and R. Shearer, “The open world assumption,” in eSI Workshop
2006
Earlier work this paper cites.
T. Fawcett, “An introduction to roc analysis,” Pattern recognition letters
2006
Earlier work this paper cites.
CRC Press, 2006
D. Gamerman and H. F. Lopes, Markov chain Monte Carlo: stochastic simulation for Bayesian inference · 2006
Earlier work this paper cites.
S. Basu and M. Meckesheimer, “Automatic outlier detection for time series: an application to sensor data,” Knowledge and Information Systems
2007
Earlier work this paper cites.
G. Münz, S. Li, and G. Carle, “Traffic anomaly detection using k-means clustering,” in GI/ITG Workshop MMBnet
2007
Earlier work this paper cites.
Y. Kou, C.-T. Lu, and R. F. Dos Santos, “Spatial outlier detection: a graph-based approach,” in 19th IEEE International Conference on Tools with Artificial Intelligence (ICTAI)
2007
Earlier work this paper cites.
K. A. Nixon, V. Aimale, and R. K. Rowe, “Spoof detection schemes,” in Handbook of biometrics
2008
Earlier work this paper cites.
P. L. Bartlett and M. H. Wegkamp, “Classification with a reject option using a hinge loss.,” Journal of Machine Learning Research
2008
Earlier work this paper cites.
G. Shafer and V. Vovk, “A tutorial on conformal prediction.,” Journal of Machine Learning Research
2008
Earlier work this paper cites.
A. Gelman, “Objections to bayesian statistics,” Bayesian Analysis
2008
Earlier work this paper cites.
B. Zhang and W. Zuo, “Learning from positive and unlabeled examples: A survey,” in International Symposiums on Information Processing
2008
Earlier work this paper cites.
F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation forest,” in ICDM
2008
Earlier work this paper cites.
A. Torralba, R. Fergus, and W. T. Freeman, “80 million tiny images: A large data set for nonparametric object and scene recognition,” TPAMI
2008
Earlier work this paper cites.
Mit Press, 2009
J. Quiñonero-Candela, M. Sugiyama, N. D. Lawrence, and A. Schwaighofer, Dataset shift in machine learning · 2009
Earlier work this paper cites.
V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” ACM computing surveys (CSUR)
2009
Earlier work this paper cites.
G. Polatkan, S. Jafarpour, A. Brasoveanu, S. Hughes, and I. Daubechies, “Detection of forgery in paintings using supervised learning,” in ICIP
2009
Earlier work this paper cites.
A. Kind, M. P. Stoecklin, and X. Dimitropoulos, “Histogram-based traffic anomaly detection,” IEEE Transactions on Network and Service Management
2009
Earlier work this paper cites.
X. Yang, L. J. Latecki, and D. Pokrajac, “Outlier detection with globally optimal exemplar-based gmm,” in SIAM
2009
Earlier work this paper cites.
A. Krizhevsky, G. Hinton, et al
2009
Earlier work this paper cites.
A. Krizhevsky, V. Nair, and G. Hinton, “Cifar-10 and cifar-100 datasets,” URl: https://www. cs. toronto. edu/kriz/cifar. html
2009
Earlier work this paper cites.
S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, “A theory of learning from different domains,” Machine learning
2010
Earlier work this paper cites.
D. Miljković, “Review of novelty detection methods,” in MIPRO
2010
Earlier work this paper cites.
L.-J. Li and L. Fei-Fei, “Optimol: automatic online picture collection via incremental model learning,” IJCV
2010
Earlier work this paper cites.
E. Sorio, A. Bartoli, G. Davanzo, and E. Medvet, “Open world classification of printed invoices,” in Proceedings of the 10th ACM symposium on Document engineering
2010
Earlier work this paper cites.
G. H. Orair, C. H. Teixeira, W. Meira Jr, Y. Wang, and S. Parthasarathy, “Distance-based outlier detection: consolidation and renewed bearing,” Proceedings of the VLDB Endowment
2010
Earlier work this paper cites.
W. Liu, J. He, and S.-F. Chang, “Large graph construction for scalable semi-supervised learning,” in ICML
2010
Earlier work this paper cites.
J. Ngiam, Z. Chen, P. W. Koh, and A. Y. Ng, “Learning deep energy models,” in ICML
2011
Earlier work this paper cites.
M. Welling and Y. W. Teh, “Bayesian learning via stochastic gradient langevin dynamics,” in ICML
2011
Earlier work this paper cites.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011
2011
Earlier work this paper cites.
G. Kylberg, “Kylberg texture dataset v. 1.0,” 2011
2011
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in NIPS
2012
Earlier work this paper cites.
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in CVPR
2012
Earlier work this paper cites.
Elsevier, 2012
E. Castillo, Extreme value theory in engineering · 2012
Earlier work this paper cites.
M. Xie, J. Hu, and B. Tian, “Histogram-based online anomaly detection in hierarchical wireless sensor networks,” in ICTSPCC
2012
Earlier work this paper cites.
M. Goldstein and A. Dengel, “Histogram-based outlier score (hbos): A fast unsupervised anomaly detection algorithm,” KI-2012: Poster and Demo Track
2012
Earlier work this paper cites.
I. Syarif, A. Prugel-Bennett, and G. Wills, “Unsupervised clustering approach for network anomaly detection,” in International conference on networked digital technologies
2012
Earlier work this paper cites.
Z. Mingqiang, H. Hui, and W. Qian, “A graph-based clustering algorithm for anomaly intrusion detection,” in International Conference on Computer Science & Education (ICCSE)
2012
Earlier work this paper cites.
W. J. Scheirer, A. de Rezende Rocha, A. Sapkota, and T. E. Boult, “Toward open set recognition,” TPAMI
2013
Earlier work this paper cites.
P. Bodesheim, A. Freytag, E. Rodner, M. Kemmler, and J. Denzler, “Kernel null space methods for novelty detection,” in CVPR
2013
Earlier work this paper cites.
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114
Original
2013
Earlier work this paper cites.
X. Mo, V. Monga, R. Bala, and Z. Fan, “Adaptive sparse representations for video anomaly detection,” IEEE Transactions on Circuits and Systems for Video Technology
2013
Earlier work this paper cites.
Y. Xiao, H. Wang, W. Xu, and J. Zhou, “L1 norm based kpca for novelty detection,” Pattern Recognition
2013
Earlier work this paper cites.
C. Leys, C. Ley, O. Klein, P. Bernard, and L. Licata, “Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median,” Journal of experimental social psychology
2013
Earlier work this paper cites.
M. Sugiyama and K. Borgwardt, “Rapid distance-based outlier detection via sampling,” NIPS
2013
Earlier work this paper cites.
M. A. Pimentel, D. A. Clifton, L. Clifton, and L. Tarassenko, “A review of novelty detection,” Signal Processing
2014
Earlier work this paper cites.
L. P. Jain, W. J. Scheirer, and T. E. Boult, “Multi-class open set recognition using probability of inclusion,” in ECCV
2014
Earlier work this paper cites.
W. J. Scheirer, L. P. Jain, and T. E. Boult, “Probability models for open set recognition,” TPAMI
2014
Earlier work this paper cites.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” NIPS
2014
Earlier work this paper cites.
J. Tian, M. H. Azarian, and M. Pecht, “Anomaly detection using self-organizing maps-based k-nearest neighbor algorithm,” in PHM Society European Conference
2014
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in ICCV
2015
Earlier work this paper cites.
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in ICLR
2015
Earlier work this paper cites.
D. Wen, H. Han, and A. K. Jain, “Face spoof detection with image distortion analysis,” IEEE Transactions on Information Forensics and Security
2015
Earlier work this paper cites.
H. Al-Behadili, A. Grumpe, and C. Wöhler, “Incremental learning and novelty detection of gestures in a multi-class system,” in AIMS
2015
Earlier work this paper cites.
Y. Xia, X. Cao, F. Wen, G. Hua, and J. Sun, “Learning discriminative reconstructions for unsupervised outlier removal,” in CVPR
2015
Earlier work this paper cites.
T. Xiao, C. Zhang, and H. Zha, “Learning to detect anomalies in surveillance video,” IEEE Signal Processing Letters
2015
Earlier work this paper cites.
X. Chen and A. Gupta, “Webly supervised learning of convolutional networks,” in ICCV
2015
Earlier work this paper cites.
A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in CVPR
2015
Earlier work this paper cites.
A. Bendale and T. Boult, “Towards open world recognition,” in CVPR
2015
Earlier work this paper cites.
D. Rezende and S. Mohamed, “Variational inference with normalizing flows,” in ICML
2015
Earlier work this paper cites.
A. Adler, M. Elad, Y. Hel-Or, and E. Rivlin, “Sparse coding with anomaly detection,” Journal of Signal Processing Systems
2015
Earlier work this paper cites.
J. An and S. Cho, “Variational autoencoder based anomaly detection using reconstruction probability,” Special Lecture on IE
2015
Earlier work this paper cites.
L. Akoglu, H. Tong, and D. Koutra, “Graph based anomaly detection and description: a survey,” Data mining and knowledge discovery
2015
Earlier work this paper cites.
D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané, “Concrete problems in AI safety,” arXiv preprint arXiv:1606.06565
Original
2016
Earlier work this paper cites.
L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in CVPR
2016
Earlier work this paper cites.
K. Patel, H. Han, and A. K. Jain, “Secure face unlock: Spoof detection on smartphones,” IEEE transactions on information forensics and security
2016
Earlier work this paper cites.
M. Bianchini, A. Belahcen, and F. Scarselli, “A comparative study of inductive and transductive learning with feedforward neural networks,” in Conference of the Italian Association for Artificial Intelligence
2016
Earlier work this paper cites.
Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in ICML
2016
Earlier work this paper cites.
A. Van Oord, N. Kalchbrenner, and K. Kavukcuoglu, “Pixel recurrent neural networks,” in ICML
2016
Earlier work this paper cites.
A. Bendale and T. E. Boult, “Towards open set deep networks,” in CVPR
2016
Earlier work this paper cites.
H. Zhang and V. M. Patel, “Sparse representation-based open set recognition,” TPAMI
2016
Earlier work this paper cites.
M. Turcotte, J. Moore, N. Heard, and A. McPhall, “Poisson factorization for peer-based anomaly detection,” in IEEE Conference on Intelligence and Security Informatics (ISI)
2016
Earlier work this paper cites.
S. Zhai, Y. Cheng, W. Lu, and Z. Zhang, “Deep structured energy based models for anomaly detection,” in ICML
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR
2016
Earlier work this paper cites.
D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in ICLR
2017
Earlier work this paper cites.
D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, “Deeper, broader and artier domain generalization,” in ICCV
2017
Earlier work this paper cites.
D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in ICLR
2017
Earlier work this paper cites.
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self-supervised prediction,” in ICML
2017
Earlier work this paper cites.
O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. Küttler, J. Agapiou, J. Schrittwieser, et al
Original
2017
Earlier work this paper cites.
T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” arXiv preprint arXiv:1708.04552
Original
2017
Earlier work this paper cites.
B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” NeurIPS
2017
Earlier work this paper cites.
E. M. Rudd, L. P. Jain, W. J. Scheirer, and T. E. Boult, “The extreme value machine,” IEEE transactions on pattern analysis and machine intelligence
2017
Earlier work this paper cites.
Z. Ge, S. Demyanov, Z. Chen, and R. Garnavi, “Generative openmax for multi-class open set classification,” in BMVC
2017
Earlier work this paper cites.
P. R. M. Júnior, R. M. De Souza, R. d. O. Werneck, B. V. Stein, D. V. Pazinato, W. R. de Almeida, O. A. Penatti, R. d. S. Torres, and A. Rocha, “Nearest neighbors distance ratio open-set classifier,” Machine Learning
2017
Earlier work this paper cites.
J. Liu, Z. Lian, Y. Wang, and J. Xiao, “Incremental kernel null space discriminant analysis for novelty detection,” in CVPR
2017
Earlier work this paper cites.
A. Li, Z. Miao, Y. Cen, and Y. Cen, “Anomaly detection using sparse reconstruction in crowded scenes,” Multimedia Tools and Applications
2017
Earlier work this paper cites.
Y. Li, J. Yang, Y. Song, L. Cao, J. Luo, and L.-J. Li, “Learning from noisy labels with distillation,” in CVPR
2017
Earlier work this paper cites.

Similar

B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence
2017
Cited alongside, same era.
P. Panareda Busto and J. Gall, “Open set domain adaptation,” in ICCV
2017
Cited alongside, same era.
M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” Neurocomputing
2018
Cited alongside, same era.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” ICLR
2018
Cited alongside, same era.
N. Akhtar and A. Mian, “Threat of adversarial attacks on deep learning in computer vision: A survey,” IEEE Access
2018
Cited alongside, same era.
D. J. Atha and M. R. Jahanshahi, “Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection,” Structural Health Monitoring
2018
Cited alongside, same era.
H. Idrees, M. Shah, and R. Surette, “Enhancing camera surveillance using computer vision: a research note,” Policing: An International Journal
2018
Cited alongside, same era.
A. R. Dhamija, M. Günther, and T. E. Boult, “Reducing network agnostophobia,” in NeurIPS
2018
Cited alongside, same era.
S. Liu, R. Garrepalli, T. Dietterich, A. Fern, and D. Hendrycks, “Open category detection with pac guarantees,” in ICML
2018
Cited alongside, same era.
Y. Wang, W. Liu, X. Ma, J. Bailey, H. Zha, L. Song, and S.-T. Xia, “Iterative learning with open-set noisy labels,” in CVPR
2018
Cited alongside, same era.
S. Liang, Y. Li, and R. Srikant, “Enhancing the reliability of out-of-distribution image detection in neural networks,” in ICLR
2018
Cited alongside, same era.
K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” in NeurIPS
2018
Cited alongside, same era.

Then

G. Kwon, M. Prabhushankar, D. Temel, and G. AlRegib, “Backpropagated gradient representations for anomaly detection,” in ECCV
2020
Later among the works it cites.
J. Bekker and J. Davis, “Learning from positive and unlabeled data: A survey,” Machine Learning
2020
Later among the works it cites.
L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft, “Deep semi-supervised anomaly detection,” ICLR
2020
Later among the works it cites.
L. Bergman and Y. Hoshen, “Classification-based anomaly detection for general data,” in ICLR
2020
Later among the works it cites.
J. Yang, W. Chen, L. Feng, X. Yan, H. Zheng, and W. Zhang, “Webly supervised image classification with metadata: Automatic noisy label correction via visual-semantic graph,” in ACM Multimedia
2020
Later among the works it cites.
D. T. Nguyen, C. K. Mummadi, T. P. N. Ngo, T. H. P. Nguyen, L. Beggel, and T. Brox, “Self: Learning to filter noisy labels with self-ensembling,” in ICLR
2020
Later among the works it cites.
J. Yang, L. Feng, W. Chen, X. Yan, H. Zheng, P. Luo, and W. Zhang, “Webly supervised image classification with self-contained confidence,” in ECCV
2020
Later among the works it cites.
C. Zhou, G. Neubig, J. Gu, M. Diab, P. Guzman, L. Zettlemoyer, and M. Ghazvininejad, “Detecting hallucinated content in conditional neural sequence generation,” ACL
2020
Later among the works it cites.
S. Mohseni, H. Wang, Z. Yu, C. Xiao, Z. Wang, and J. Yadawa, “Practical machine learning safety: A survey and primer,” arXiv preprint arXiv:2106.04823
Original
2021
Closest in time.
D. Hendrycks, N. Carlini, J. Schulman, and J. Steinhardt, “Unsolved problems in ML safety,” arXiv preprint arXiv:2109.13916
Original
2021
Closest in time.
L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller, “A unifying review of deep and shallow anomaly detection,” Proceedings of the IEEE
2021
Closest in time.
A. Mahdavi and M. Carvalho, “A survey on open set recognition,” arXiv preprint arXiv:2109.00893
Original
2021
Closest in time.
L. Zhang, M. Goldstein, and R. Ranganath, “Understanding failures in out-of-distribution detection with deep generative models,” in ICML
2021
Closest in time.
L. Jiang, Z. Guo, W. Wu, Z. Liu, Z. Liu, C. C. Loy, S. Yang, Y. Xiong, W. Xia, B. Chen, P. Zhuang, S. Li, S. Chen, T. Yao, S. Ding, J. Li, F. Huang, L. Cao, R. Ji, C. Lu, and G. Tan, “DeeperForensics Challenge 2020 on real-world face forgery detection: Methods and results,” arXiv preprint arXiv:2102.09471
Original
2021
Closest in time.
Z. Fang, J. Lu, A. Liu, F. Liu, and G. Zhang, “Learning bounds for open-set learning,” in ICML
2021
Closest in time.
[Online; accessed 12 August 2021]
Wikipedia contributors, “Outlier from Wikipedia, the free encyclopedia,” 2021 · 2021
Closest in time.
K. Cao, M. Brbic, and J. Leskovec, “Open-world semi-supervised learning,” arXiv preprint arXiv:2102.03526
Original
2021
Closest in time.
K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” arXiv preprint arXiv:2103.02503
Original
2021
Closest in time.
B. Zhao and K. Han, “Novel visual category discovery with dual ranking statistics and mutual knowledge distillation,” NeurIPS
2021
Closest in time.
X. Jia, K. Han, Y. Zhu, and B. Green, “Joint representation learning and novel category discovery on single-and multi-modal data,” in ICCV
2021
Closest in time.
A. N. Angelopoulos and S. Bates, “A gentle introduction to conformal prediction and distribution-free uncertainty quantification,” arXiv preprint arXiv:2107.07511
Original
2021
Closest in time.
F. Cai, A. I. Ozdagli, N. Potteiger, and X. Koutsoukos, “Inductive conformal out-of-distribution detection based on adversarial autoencoders,” in 2021 IEEE International Conference on Omni-Layer Intelligent Systems (COINS)
2021
Closest in time.
M. Salehi, H. Mirzaei, D. Hendrycks, Y. Li, M. H. Rohban, and M. Sabokrou, “A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges,” arXiv preprint arXiv:2110.14051
Original
2021
Closest in time.
H. Wang, W. Liu, A. Bocchieri, and Y. Li, “Can multi-label classification networks know what they don’t know?,” NeurIPS
2021
Closest in time.
Y. Sun, C. Guo, and Y. Li, “React: Out-of-distribution detection with rectified activations,” in NeurIPS
2021
Closest in time.
Z. Lin, S. D. Roy, and Y. Li, “Mood: Multi-level out-of-distribution detection,” in CVPR
2021
Closest in time.
Y. Wang, B. Li, T. Che, K. Zhou, Z. Liu, and D. Li, “Energy-based open-world uncertainty modeling for confidence calibration,” in ICCV
2021
Closest in time.
J. Chen, Y. Li, X. Wu, Y. Liang, and S. Jha, “Atom: Robustifying out-of-distribution detection using outlier mining,” ECML&PKDD
2021
Closest in time.
K. Bibas, M. Feder, and T. Hassner, “Single layer predictive normalized maximum likelihood for out-of-distribution detection,” NeurIPS
2021
Closest in time.
R. Huang and Y. Li, “Mos: Towards scaling out-of-distribution detection for large semantic space,” in CVPR
2021
Closest in time.
S. Fort, J. Ren, and B. Lakshminarayanan, “Exploring the limits of out-of-distribution detection,” NeurIPS
2021
Closest in time.
W. Gan, “Language guided out-of-distribution detection,” 2021
2021
Closest in time.
S. Thulasidasan, S. Thapa, S. Dhaubhadel, G. Chennupati, T. Bhattacharya, and J. Bilmes, “An effective baseline for robustness to distributional shift,” arXiv preprint arXiv:2105.07107
Original
2021
Closest in time.
A.-A. Papadopoulos, M. R. Rajati, N. Shaikh, and J. Wang, “Outlier exposure with confidence control for out-of-distribution detection,” Neurocomputing
2021
Closest in time.
J. Yang, H. Wang, L. Feng, X. Yan, H. Zheng, W. Zhang, and Z. Liu, “Semantically coherent out-of-distribution detection,” in ICCV
2021
Closest in time.
R. Huang, A. Geng, and Y. Li, “On the importance of gradients for detecting distributional shifts in the wild,” in NeurIPS
2021
Closest in time.
K. Kim, J. Shin, and H. Kim, “Locally most powerful bayesian test for out-of-distribution detection using deep generative models,” NeurIPS
2021
Closest in time.
D. Jiang, S. Sun, and Y. Yu, “Revisiting flow generative models for out-of-distribution detection,” in International Conference on Learning Representations
2021
Closest in time.
J. Ren, S. Fort, J. Liu, A. G. Roy, S. Padhy, and B. Lakshminarayanan, “A simple fix to mahalanobis distance for improving near-ood detection,” arXiv preprint arXiv:2106.09022
Original
2021
Closest in time.
A. Zaeemzadeh, N. Bisagno, Z. Sambugaro, N. Conci, N. Rahnavard, and M. Shah, “Out-of-distribution detection using union of 1-dimensional subspaces,” in CVPR
2021
Closest in time.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al
2021
Closest in time.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al
Original
2021
Closest in time.
X. Sun, H. Ding, C. Zhang, G. Lin, and K.-V. Ling, “M2iosr: Maximal mutual information open set recognition,” arXiv preprint arXiv:2108.02373
Original
2021
Closest in time.
D.-W. Zhou, H.-J. Ye, and D.-C. Zhan, “Learning placeholders for open-set recognition,” in CVPR
2021
Closest in time.
S. Kong and D. Ramanan, “Opengan: Open-set recognition via open data generation,” in ICCV
2021
Closest in time.
Z. Yue, T. Wang, Q. Sun, X.-S. Hua, and H. Zhang, “Counterfactual zero-shot and open-set visual recognition,” in CVPR
2021
Closest in time.
G. Chen, P. Peng, L. Ma, J. Li, L. Du, and Y. Tian, “Amplitude-phase recombination: Rethinking robustness of convolutional neural networks in frequency domain,” ICCV
2021
Closest in time.
H. Liu, X. Li, W. Zhou, Y. Chen, Y. He, H. Xue, W. Zhang, and N. Yu, “Spatial-phase shallow learning: rethinking face forgery detection in frequency domain,” in CVPR
2021
Closest in time.
K. Jiang, W. Xie, J. Lei, T. Jiang, and Y. Li, “Lren: Low-rank embedded network for sample-free hyperspectral anomaly detection,” in AAAI
2021
Closest in time.
X. Yan, H. Zhang, X. Xu, X. Hu, and P.-A. Heng, “Learning semantic context from normal samples for unsupervised anomaly detection,” in AAAI
2021
Closest in time.
M.-I. Georgescu, A. Barbalau, R. T. Ionescu, F. S. Khan, M. Popescu, and M. Shah, “Anomaly detection in video via self-supervised and multi-task learning,” in CVPR
2021
Closest in time.
Z.-F. Wu, T. Wei, J. Jiang, C. Mao, M. Tang, and Y.-F. Li, “Ngc: A unified framework for learning with open-world noisy data,” in ICCV
2021
Closest in time.
P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, et al
2021
Closest in time.
Y. Shu, Z. Cao, C. Wang, J. Wang, and M. Long, “Open domain generalization with domain-augmented meta-learning,” in CVPR
2021
Closest in time.
J. Li, C. Xiong, and S. C. Hoi, “Mopro: Webly supervised learning with momentum prototypes,” ICLR
2021
Closest in time.
D. Hendrycks and M. Mazeika, “X-risk analysis for AI research,” arXiv preprint arXiv:2206.05862
Original
2022
Closest in time.
D. Hendrycks, S. Basart, M. Mazeika, M. Mostajabi, J. Steinhardt, and D. Song, “Scaling out-of-distribution detection for real-world settings,” in ICML
2022
Closest in time.
D. Zimmerer, P. M. Full, F. Isensee, P. Jäger, T. Adler, J. Petersen, G. Köhler, T. Ross, A. Reinke, A. Kascenas, et al
2022
Closest in time.
S. Vaze, K. Han, A. Vedaldi, and A. Zisserman, “Generalized category discovery,” in CVPR
2022
Closest in time.
K. Joseph, S. Paul, G. Aggarwal, S. Biswas, P. Rai, K. Han, and V. N. Balasubramanian, “Novel class discovery without forgetting,” in ECCV
2022
Closest in time.
R. Kaur, S. Jha, A. Roy, S. Park, E. Dobriban, O. Sokolsky, and I. Lee, “idecode: In-distribution equivariance for conformal out-of-distribution detection,” in Proceedings of the AAAI Conference on Artificial Intelligence
2022
Closest in time.
R. Kaur, K. Sridhar, S. Park, S. Jha, A. Roy, O. Sokolsky, and I. Lee, “Codit: Conformal out-of-distribution detection in time-series data,” arXiv e-prints
2022
Closest in time.
X. Dong, J. Guo, A. Li, W.-T. Ting, C. Liu, and H. Kung, “Neural mean discrepancy for efficient out-of-distribution detection,” in CVPR
2022
Closest in time.
Y. Sun and Y. Li, “Dice: Leveraging sparsification for out-of-distribution detection,” in ECCV
2022
Closest in time.
Y. Sun, Y. Ming, X. Zhu, and Y. Li, “Out-of-distribution detection with deep nearest neighbors,” in ICML
2022
Closest in time.
D. Hendrycks, A. Zou, M. Mazeika, L. Tang, D. Song, and J. Steinhardt, “Pixmix: Dreamlike pictures comprehensively improve safety measures,” 2022
2022
Closest in time.
Q. Wang, F. Liu, Y. Zhang, J. Zhang, C. Gong, T. Liu, and B. Han, “Watermarking for out-of-distribution detection,” in NeurIPS
2022
Closest in time.
X. Dong, J. Guo, W.-T. T. Ang Li23, C. Liu, and H. Kung, “Neural mean discrepancy for efficient out-of-distribution detection,” in CVPR
2022
Closest in time.
H. Wei, R. Xie, H. Cheng, L. Feng, B. An, and Y. Li, “Mitigating neural network overconfidence with logit normalization,” in ICML
2022
Closest in time.
Y. Ming, Y. Fan, and Y. Li, “Poem: Out-of-distribution detection with posterior sampling,” in ICML
2022
Closest in time.
J. Katz-Samuels, J. Nakhleh, R. Nowak, and Y. Li, “Training ood detectors in their natural habitats,” in International Conference on Machine Learning (ICML)
2022
Closest in time.
X. Du, Z. Wang, M. Cai, and Y. Li, “Vos: Learning what you don’t know by virtual outlier synthesis,” in Proceedings of the International Conference on Learning Representations
2022
Closest in time.
X. Du, X. Wang, G. Gozum, and Y. Li, “Unknown-aware object detection: Learning what you don’t know from videos in the wild,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
2022
Closest in time.
C. Igoe, Y. Chung, I. Char, and J. Schneider, “How useful are gradients for ood detection really?,” arXiv preprint arXiv:2205.10439
Original
2022
Closest in time.
S. Esmaeilpour, B. Liu, E. Robertson, and L. Shu, “Zero-shot out-of-distribution detection based on the pretrained model clip,” in AAAI
2022
Closest in time.
Y. Ming, Z. Cai, J. Gu, Y. Sun, W. Li, and Y. Li, “Delving into out-of-distribution detection with vision-language representations,” Advances in Neural Information Processing Systems
2022
Closest in time.
H. Wang, Z. Li, L. Feng, and W. Zhang, “Vim: Out-of-distribution with virtual-logit matching,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
2022
Closest in time.
Y. Zhou, “Rethinking reconstruction autoencoder-based out-of-distribution detection,” in CVPR
2022
Closest in time.
Y. Yang, R. Gao, and Q. Xu, “Out-of-distribution detection with semantic mismatch under masking,” in ECCV
2022
Closest in time.
P. Morteza and Y. Li, “Provable guarantees for understanding out-of-distribution detection,” in AAAI
2022
Closest in time.
Z. Fang, Y. Li, J. Lu, J. Dong, B. Han, and F. Liu, “Is out-of-distribution detection learnable?,” in NeurIPS
2022
Closest in time.
K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” International Journal of Computer Vision (IJCV)
2022
Closest in time.
K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Conditional prompt learning for vision-language models,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
2022
Closest in time.
M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” in European Conference on Computer Vision
2022
Closest in time.
J. Yang, K. Zhou, and Z. Liu, “Full-spectrum out-of-distribution detection,” arXiv preprint arXiv:2204.05306
Original
2022
Closest in time.
E. D. C. Gomes, F. Alberge, P. Duhamel, and P. Piantanida, “Igeood: An information geometry approach to out-of-distribution detection,” in ICLR
2022
Closest in time.
S. Vaze, K. Han, A. Vedaldi, and A. Zisserman, “Open-set recognition: A good closed-set classifier is all you need,” in ICLR
2022
Closest in time.
J. Yang, P. Wang, D. Zou, Z. Zhou, K. Ding, W. Peng, H. Wang, G. Chen, B. Li, Y. Sun, X. Du, K. Zhou, W. Zhang, D. Hendrycks, Y. Li, and Z. Liu, “Openood: Benchmarking generalized out-of-distribution detection,” in NeurIPS
2022
Closest in time.
Y. Ming, H. Yin, and Y. Li, “On the impact of spurious correlation for out-of-distribution detection,” in AAAI
2022
Closest in time.
V. D. Nguyen, “Out-of-distribution detection for lidar-based 3d object detection,” Master’s thesis, University of Waterloo, 2022
2022
Closest in time.
G. Shalev, G.-L. Shalev, and J. Keshet, “A baseline for detecting out-of-distribution examples in image captioning,” arXiv preprint arXiv:2207.05418
Original
2022
Closest in time.
J. Zhang, N. Inkawhich, R. Linderman, Y. Chen, and H. Li, “Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
2023
Closest in time.
R. Averly and W.-L. Chao, “Unified out-of-distribution detection: A model-specific perspective,” arXiv preprint arXiv:2304.06813
Original
2023
Closest in time.
J. Parmar, S. Chouhan, V. Raychoudhury, and S. Rathore, “Open-world machine learning: applications, challenges, and opportunities,” ACM Computing Surveys
2023
Closest in time.
J. Zhang, Q. Fu, X. Chen, L. Du, Z. Li, G. Wang, S. Han, D. Zhang, et al
2023
Closest in time.
A. Djurisic, N. Bozanic, A. Ashok, and R. Liu, “Extremely simple activation shaping for out-of-distribution detection,” ICLR
2023
Closest in time.
J. Park, Y. G. Jung, and A. B. J. Teoh, “Nearest neighbor guidance for out-of-distribution detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision
2023
Closest in time.
J. Park, J. C. L. Chai, J. Yoon, and A. B. J. Teoh, “Understanding the feature norm for out-of-distribution detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision
2023
Closest in time.
X. Jiang, F. Liu, Z. Fang, H. Chen, T. Liu, F. Zheng, and B. Han, “Detecting out-of-distribution data through in-distribution class prior,” in International Conference on Machine Learning
2023
Closest in time.
X. Liu, Y. Lochman, and C. Zach, “Gen: Pushing the limits of softmax-based out-of-distribution detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
2023
Closest in time.
R. Linderman, J. Zhang, N. Inkawhich, H. Li, and Y. Chen, “Fine-grain inference on out-of-distribution data with hierarchical classification,” in Proceedings of The 2nd Conference on Lifelong Learning Agents
2023
Closest in time.
F. Lu, K. Zhu, W. Zhai, K. Zheng, and Y. Cao, “Uncertainty-aware optimal transport for semantically coherent out-of-distribution detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
2023
Closest in time.
Q. Wang, Z. Fang, Y. Zhang, F. Liu, Y. Li, and B. Han, “Learning to augment distributions for out-of-distribution detection,” arXiv preprint arXiv:2311.01796
Original
2023
Closest in time.
L. Tao, X. Du, X. Zhu, and Y. Li, “Non-parametric outlier synthesis,” in ICLR
2023
Closest in time.
Q. Wang, J. Ye, F. Liu, Q. Dai, M. Kalander, T. Liu, J. Hao, and B. Han, “Out-of-distribution detection with implicit outlier transformation,” 2023
2023
Closest in time.
H. Zheng, Q. Wang, Z. Fang, X. Xia, F. Liu, T. Liu, and B. Han, “Out-of-distribution detection learning with unreliable out-of-distribution sources,” in NeurIPS
2023
Closest in time.
Y. Ming and Y. Li, “How does fine-tuning impact out-of-distribution detection for vision-language models?,” IJCV
2023
Closest in time.
A. Miyai, Q. Yu, G. Irie, and K. Aizawa, “Can pre-trained networks detect familiar out-of-distribution data?,” arXiv preprint arXiv:2310.00847
Original
2023
Closest in time.
A. Miyai, Q. Yu, G. Irie, and K. Aizawa, “Locoop: Few-shot out-of-distribution detection via prompt learning,” arXiv preprint arXiv:2306.01293
Original
2023
Closest in time.
F. Lu, K. Zhu, K. Zheng, W. Zhai, and Y. Cao, “Likelihood-aware semantic alignment for full-spectrum out-of-distribution detection,” arXiv preprint arXiv:2312.01732
Original
2023
Closest in time.
H. Wang, Y. Li, H. Yao, and X. Li, “Clipn for zero-shot ood detection: Teaching clip to say no,” in Proceedings of the IEEE/CVF International Conference on Computer Vision
2023
Closest in time.
Y. Ming, Y. Sun, O. Dia, and Y. Li, “Cider: Exploiting hyperspherical embeddings for out-of-distribution detection,” in ICLR
2023
Closest in time.
J.-H. Kim, S. Yun, and H. O. Song, “Neural relation graph: A unified framework for identifying label noise and outlier data,” in Thirty-seventh Conference on Neural Information Processing Systems
2023
Closest in time.
W. Jiang, H. Cheng, M. Chen, S. Feng, Y. Ge, and C. Wang, “Read: Aggregating reconstruction error into out-of-distribution detection,” in AAAI
2023
Closest in time.
J. Li, P. Chen, S. Yu, Z. He, S. Liu, and J. Jia, “Rethinking out-of-distribution (ood) detection: Masked image modeling is all you need,” in CVPR
2023
Closest in time.
P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao, “Clip-adapter: Better vision-language models with feature adapters,” International Journal of Computer Vision
2023
Closest in time.
J. Zhang, J. Yang, P. Wang, H. Wang, Y. Lin, H. Zhang, Y. Sun, X. Du, K. Zhou, W. Zhang, Y. Li, Z. Liu, Y. Chen, and H. Li, “Openood v1.5: Enhanced benchmark for out-of-distribution detection,” arXiv preprint arXiv:2306.09301
Original
2023
Closest in time.
J. Bitterwolf, M. Müller, and M. Hein, “In or out? fixing imagenet out-of-distribution detection evaluation,” in ICML
2023
Closest in time.
L. Cultrera, L. Seidenari, and A. Del Bimbo, “Leveraging visual attention for out-of-distribution detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision
2023
Closest in time.
X. Wu, J. Lu, Z. Fang, and G. Zhang, “Meta ood learning for continuously adaptive ood detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision
2023
Closest in time.
Y. Dai, H. Lang, K. Zeng, F. Huang, and Y. Li, “Exploring large language models for multi-modal out-of-distribution detection,” arXiv preprint arXiv:2310.08027
Original
2023
Closest in time.
Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang, “The dawn of lmms: Preliminary explorations with gpt-4v (ision),” arXiv preprint arXiv:2309.17421
Original
2023
Closest in time.
H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” arXiv preprint arXiv:2304.08485
Original
2023
Closest in time.
B. Li, Y. Zhang, L. Chen, J. Wang, J. Yang, and Z. Liu, “Otter: A multi-modal model with in-context instruction tuning,” arXiv preprint arXiv:2305.03726
Original
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…