2023

In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation

Bitterwolf, Julian, Müller, Maximilian, Hein, Matthias

Understand

Out-of-distribution (OOD) detection is the problem of identifying inputs which are unrelated to the in-distribution task.

The OOD detection performance when the in-distribution (ID) is ImageNet-1K is commonly being tested on a small range of test OOD datasets.
We find that most of the currently used test OOD datasets, including datasets from the open set recognition (OSR) literature, have severe issues: In some cases more than 50$\%$ of the dataset contains objects belonging to one of the ID classes.
These erroneous samples heavily distort the evaluation of OOD detectors.

Built on

Wordnet: An electronic lexical database
Fellbaum, C · 1998
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A. and Hinton, G · 2009
Earlier work this paper cites.
Food-101 – mining discriminative components with random forests
Bossard, L., Guillaumin, M., and Van Gool, L · 2014
Earlier work this paper cites.
Describing textures in the wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A · 2014
Earlier work this paper cites.
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L · 2015
Earlier work this paper cites.
Towards open set deep networks
Bendale, A. and Boult, T. E · 2016
Earlier work this paper cites.
A baseline for detecting misclassified and out-of-distribution examples in neural networks
Hendrycks, D. and Gimpel, K · 2017
Earlier work this paper cites.
Openimages: A public dataset for large-scale multi-label and multi-class image classification
Krasin, I., Duerig, T., Alldrin, N., Ferrari, V., Abu-El-Haija, S., Kuznetsova, A., Rom, H., Uijlings, J., Popov, S., Kamali, S., Malloci, M., Pont-Tuset, J., Veit, A., Belongie, S., Gomes, V., Gupta, A., Sun, C., Chechik, G., Cai, D., Feng, Z., Narayanan, D., and Murphy, K · 2017
Earlier work this paper cites.
Places: A 10 million image database for scene recognition
Original
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A · 2017
Earlier work this paper cites.
A simple unified framework for detecting out-of-distribution samples and adversarial attacks
Lee, K., Lee, K., Lee, H., and Shin, J · 2018
Earlier work this paper cites.
The inaturalist species classification and detection dataset
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S · 2018
Earlier work this paper cites.
Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem
Hein, M., Andriushchenko, M., and Bitterwolf, J · 2019
Earlier work this paper cites.
Deep anomaly detection with outlier exposure
Hendrycks, D., Mazeika, M., and Dietterich, T · 2019
Earlier work this paper cites.
Pytorch image models
Wightman, R · 2019
Earlier work this paper cites.
Self-training with noisy student improves imagenet classification
Original
Xie, Q., Luong, M.-T., Hovy, E., and Le, Q. V · 2019
Earlier work this paper cites.
Certifiably adversarially robust detection of out-of-distribution data
Bitterwolf, J., Meinke, A., and Hein, M · 2020
Earlier work this paper cites.

Similar

Mynursinghome: A fully-labelled image dataset for indoor object classification
Ismail, A., Ahmad, S. A., Che Soh, A., Hassan, M. K., and Harith, H. H · 2020
Cited alongside, same era.
Big transfer (bit): General visual representation learning
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N · 2020
Cited alongside, same era.
Energy-based out-of-distribution detection
Liu, W., Wang, X., Owens, J., and Li, Y · 2020
Cited alongside, same era.
Csi: Novelty detection via contrastive learning on distributionally shifted instances
Tack, J., Mo, S., Jeong, J., and Shin, J · 2020
Cited alongside, same era.
Hyperparameter-free out-of-distribution detection using cosine similarity
Techapanurak, E., Suganuma, M., and Okatani, T · 2020
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N · 2021
Cited alongside, same era.
Exploring the limits of out-of-distribution detection
Fort, S., Ren, J., and Lakshminarayanan, B · 2021
Cited alongside, same era.
Natural adversarial examples
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D · 2021
Cited alongside, same era.
Mos: Towards scaling out-of-distribution detection for large semantic space
Huang, R. and Li, Y · 2021
Cited alongside, same era.
Oodformer: Out-of-distribution detection transformer
Original
Koner, R., Sinhamahapatra, P., Roscher, K., Günnemann, S., and Tresp, V · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I · 2021
Cited alongside, same era.
A simple fix to mahalanobis distance for improving near-ood detection, 2021
Original
Ren, J., Fort, S., Liu, J., Roy, A. G., Padhy, S., and Lakshminarayanan, B · 2021
Cited alongside, same era.

Then

Breaking down out-of-distribution detection: Many methods based on ood training data estimate a combination of the same core quantities
Bitterwolf, J., Meinke, A., Augustin, M., and Hein, M · 2022
Later among the works it cites.
Scaling out-of-distribution detection for real-world settings
Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D · 2022
Later among the works it cites.
Caltech 101, 2022
Li, F.-F., Andreeto, M., Ranzato, M., and Perona, P · 2022
Later among the works it cites.
Provably adversarially robust detection of out-of-distribution data (almost) for free
Meinke, A., Bitterwolf, J., and Hein, M · 2022
Later among the works it cites.
Delving into out-of-distribution detection with vision-language representations
Ming, Y., Cai, Z., Gu, J., Sun, Y., Li, W., and Li, Y · 2022
Later among the works it cites.
Rankfeat: Rank-1 feature removal for out-of-distribution detection
Song, Y., Sebe, N., and Wang, W · 2022
Later among the works it cites.
How to train your vit? data, augmentation, and regularization in vision transformers
Steiner, A. P., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., and Beyer, L · 2022
Later among the works it cites.
Out-of-distribution detection with deep nearest neighbors
Sun, Y., Ming, Y., Zhu, X., and Li, Y · 2022
Later among the works it cites.
Deit iii: Revenge of the vit
Touvron, H., Cord, M., and Jegou, H · 2022
Later among the works it cites.
When does dough become a bagel? analyzing the remaining mistakes on imagenet
Vasudevan, V., Caine, B., Gontijo-Lopes, R., Fridovich-Keil, S., and Roelofs, R · 2022
Later among the works it cites.
Open-set recognition: a good closed-set classifier is all you need?
Vaze, S., Han, K., Vedaldi, A., and Zisserman, A · 2022
Later among the works it cites.
Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition
Wang, H., Zhang, A., Zhu, Y., Zheng, S., Li, M., Smola, A., and Wang, Z · 2022
Later among the works it cites.
A simple parametric classification baseline for generalized category discovery
Original
Wen, X., Zhao, B., and Qi, X · 2022
Later among the works it cites.
On the usefulness of deep ensemble diversity for out-of-distribution detection
Original
Xia, G. and Bouganis, C.-S · 2022
Later among the works it cites.
Openood: Benchmarking generalized out-of-distribution detection
Original
Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., Sun, Y., et al · 2022
Later among the works it cites.
A framework for benchmarking class-out-of-distribution detection and its application to imagenet
Galil, I., Dabbah, M., and El-Yaniv, R · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…