2021

Class-agnostic Object Detection with Multi-modal Transformer

Maaz, Muhammad, Rasheed, Hanoona, Khan, Salman et al.

Understand

What constitutes an object? This has been a long-standing question in computer vision.

Towards this goal, numerous learning-free and learning-based approaches have been developed to score objectness.
However, they generally do not scale well across new domains and novel objects.
In this paper, we advocate that existing methods lack a top-down supervision signal governed by human-understandable semantics.

Built on

Wu, K., Otoo, E., Shoshani, A.: Optimizing connected component labeling algorithms. In: Medical Imaging 2005: Image Processing. vol. 5747, pp. 1965–1976. International Society for Optics and Photonics (2005)
2005
Earlier work this paper cites.
Oliva, A., Torralba, A.: The role of context in object recognition. Trends in Cognitive Sciences 11
2007
Earlier work this paper cites.
Alexe, B., Deselaers, T., Ferrari, V.: What is an object? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 73–80. IEEE (2010)
2010
Earlier work this paper cites.
Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision 88
2010
Earlier work this paper cites.
Alexe, B., Deselaers, T., Ferrari, V.: Measuring the Objectness of Image Windows. IEEE Transactions on Pattern Analysis and Machine Intelligence 34
2012
Earlier work this paper cites.
Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3354–3361. IEEE (2012)
2012
Earlier work this paper cites.
Uijlings, J.R., Van De Sande, K.E., Gevers, T., Smeulders, A.W.: Selective Search for Object Recognition. International Journal of Computer Vision 104
2013
Earlier work this paper cites.
Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.H.: Saliency Detection via Graph-Based Manifold Ranking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3166–3173 (2013)
2013
Earlier work this paper cites.
Cheng, M.M., Zhang, Z., Lin, W.Y., Torr, P.: BING: Binarized Normed Gradients for Objectness Estimation at 300fps. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3286–3293 (2014)
2014
Earlier work this paper cites.
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common Objects in Context. In: The European Conference on Computer Vision. pp. 740–755. Springer (2014)
2014
Earlier work this paper cites.
Zitnick, C.L., Dollár, P.: Edge Boxes: Locating Object Proposals from Edges. In: The European Conference on Computer Vision. pp. 391–405. Springer (2014)
2014
Earlier work this paper cites.
Bendale, A., Boult, T.: Towards Open World Recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1893–1902 (2015)
2015
Earlier work this paper cites.
Kuo, W., Hariharan, B., Malik, J.: DeepBox: Learning Objectness with Convolutional Networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2479–2487 (2015)
2015
Earlier work this paper cites.
Pinheiro, P.O., Collobert, R., Dollár, P.: Learning to Segment Object Candidates. In: Advances in Neural Information Processing Systems (2015)
2015
Earlier work this paper cites.
Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2641–2649 (2015)
2015
Earlier work this paper cites.
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems 28
2015
Earlier work this paper cites.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision 115
2015
Earlier work this paper cites.
Shi, J., Yan, Q., Xu, L., Jia, J.: Hierarchical Image Saliency Detection on Extended CSSD. IEEE Transactions on Pattern Analysis and Machine Intelligence 38
2015
Earlier work this paper cites.
Georgakis, G., Reza, M.A., Mousavian, A., Le, P.H., Košecká, J.: Multiview RGB-D Dataset for Object Instance Detection. In: CoRR. pp. 426–434. IEEE (2016)
2016
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
2016
Earlier work this paper cites.
Pinheiro, P.O., Lin, T.Y., Collobert, R., Dollár, P.: Learning to Refine Object Segments. In: The European Conference on Computer Vision (2016)
2016
Earlier work this paper cites.
Pont-Tuset, J., Arbelaez, P., Barron, J.T., Marques, F., Malik, J.: Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39
2016
Earlier work this paper cites.
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2961–2969 (2017)
2017
Earlier work this paper cites.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123
2017
Earlier work this paper cites.
Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature Pyramid Networks for Object Detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2117–2125 (2017)
2017
Earlier work this paper cites.
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal Loss for Dense Object Detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2980–2988 (2017)
2017
Earlier work this paper cites.
Inoue, N., Furuta, R., Yamasaki, T., Aizawa, K.: Cross-Domain Weakly-Supervised Object Detection Through Progressive Domain Adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5001–5009 (2018)
2018
Earlier work this paper cites.
Skurowski, P., Abdulameer, H., Błaszczyk, J., Depta, T., Kornacki, A., Kozieł, P.: Animal Camouflage Analysis: CHAMELEON Database. Unpublished Manuscript 2
2018
Earlier work this paper cites.

Similar

Xia, G.S., Bai, X., Ding, J., Zhu, Z., Belongie, S., Luo, J., Datcu, M., Pelillo, M., Zhang, L.: DOTA: A Large-scale Dataset for Object Detection in Aerial Images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3974–3983 (2018)
2018
Cited alongside, same era.
Yan, K., Wang, X., Lu, L., Summers, R.M.: DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning. Journal of medical imaging 5
2018
Cited alongside, same era.
Zhang, Z., Liu, Y., Chen, X., Zhu, Y., Cheng, M.M., Saligrama, V., Torr, P.H.: BING++: A Fast High Quality Object Proposal Generator at 100fps. In: IEEE Transactions on Pattern Analysis and Machine Intelligence. vol. 40, pp. 1209–1223 (2018)
2018
Cited alongside, same era.
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: NAACL (2019)
2019
Cited alongside, same era.
Gupta, A., Dollar, P., Girshick, R.: LVIS: A Dataset for Large Vocabulary Instance Segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5356–5364 (2019)
2019
Cited alongside, same era.
Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6700–6709 (2019)
2019
Cited alongside, same era.
Le, T.N., Nguyen, T.V., Nie, Z., Tran, M.T., Sugimoto, A.: Anabranch network for camouflaged object segmentation. Computer Vision and Image Understanding 184
2019
Cited alongside, same era.
Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: VisualBERT: A Simple and Performant Baseline for Vision and Language. arXiv preprint arXiv:1908.03557 (2019)
Original
2019
Cited alongside, same era.
Liu, J.J., Hou, Q., Cheng, M.M., Feng, J., Jiang, J.: A Simple Pooling-Based Design for Real-Time Salient Object Detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3917–3926 (2019)
2019
Cited alongside, same era.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692 (2019)
Original
2019
Cited alongside, same era.
Lu, J., Batra, D., Parikh, D., Lee, S.: ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks . In: Advances in Neural Information Processing Systems (2019)
2019
Cited alongside, same era.
Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., Sun, J.: Objects365: A large-scale, high-quality dataset for object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8430–8439 (2019)
2019
Cited alongside, same era.

Then

Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al.: The open images dataset v4. IJCV 128
2020
Later among the works it cites.
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al.: Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. In: The European Conference on Computer Vision. pp. 121–137. Springer (2020)
2020
Later among the works it cites.
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., Pietikäinen, M.: Deep Learning for Generic Object Detection: A Survey. International Journal of Computer Vision 128
2020
Later among the works it cites.
Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-Task Vision and Language Representation Learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10437–10446 (2020)
2020
Later among the works it cites.
Misra, I., Maaten, L.v.d.: Self-Supervised Learning of Pretext-Invariant Representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6707–6717 (2020)
2020
Later among the works it cites.
Peyré, G., Cuturi, M.: Computational Optimal Transport (2020)
2020
Later among the works it cites.
Wang, X., Huang, T.E., Darrell, T., Gonzalez, J.E., Yu, F.: Frustratingly Simple Few-Shot Object Detection. arXiv preprint arXiv:2003.06957 (2020)
Original
2020
Later among the works it cites.
Xie, Q., Luong, M.T., Hovy, E., Le, Q.V.: Self-training with Noisy Student improves ImageNet classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10687–10698 (2020)
2020
Later among the works it cites.
Zhang, M., Tseng, C., Kreiman, G.: Putting visual object recognition in context. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12985–12994 (2020)
2020
Later among the works it cites.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging Properties in Self-Supervised Vision Transformers. arXiv preprint arXiv:2104.14294 (2021)
Original
2021
Closest in time.
Dai, Z., Cai, B., Lin, Y., Chen, J.: UP-DETR: Unsupervised Pre-training for Object Detection with Transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1601–1610 (2021)
2021
Closest in time.
Fan, D.P., Ji, G.P., Cheng, M.M., Shao, L.: Concealed Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
2021
Closest in time.
Jaiswal, A., Wu, Y., Natarajan, P., Natarajan, P.: Class-agnostic Object Detection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 919–928 (2021)
2021
Closest in time.
Joseph, K., Khan, S., Khan, F.S., Balasubramanian, V.N.: Towards Open World Object Detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5830–5840 (2021)
2021
Closest in time.
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: MDETR–Modulated Detection for End-to-End Multi-Modal Understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1780–1790 (2021)
2021
Closest in time.
Kim, D., Lin, T.Y., Angelova, A., Kweon, I.S., Kuo, W.: Learning Open-World Object Proposals without Learning to Classify. arXiv preprint arXiv:2108.06753 (2021)
Original
2021
Closest in time.
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning Transferable Visual Models From Natural Language Supervision. In: International Conference on Machine Learning (2021)
2021
Closest in time.
Siméoni, O., Puy, G., Vo, H.V., Roburin, S., Gidaris, S., Bursuc, A., Pérez, P., Marlet, R., Ponce, J.: Localizing Objects with Self-Supervised Transformers and no Labels. In: British Machine Vision Conference (2021)
2021
Closest in time.
Wang, W., Feiszli, M., Wang, H., Tran, D.: Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation. arXiv preprint arXiv:2104.04691 (2021)
Original
2021
Closest in time.
Xiao, T., Reed, C.J., Wang, X., Keutzer, K., Darrell, T.: Region Similarity Representation Learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)
2021
Closest in time.
Xie, E., Ding, J., Wang, W., Zhan, X., Xu, H., Sun, P., Li, Z., Luo, P.: DetCo: Unsupervised Contrastive Learning for Object Detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8392–8401 (2021)
2021
Closest in time.
Zareian, A., Rosa, K.D., Hu, D.H., Chang, S.F.: Open-Vocabulary Object Detection Using Captions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14393–14402 (2021)
2021
Closest in time.
Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In: International Conference on Machine Learning (2021)
2021
Closest in time.
Zhou, M., Zhou, L., Wang, S., Cheng, Y., Li, L., Yu, Z., Liu, J.: UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4155–4165 (2021)
2021
Closest in time.
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: Deformable Transformers for End-to-End Object Detection. In: International Conference on Learning Representations (2021)
2021
Closest in time.
Bar, A., Wang, X., Kantorov, V., Reed, C.J., Herzig, R., Chechik, G., Rohrbach, A., Darrell, T., Globerson, A.: DETReg: Unsupervised Pretraining with Region Priors for Object Detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)
2022
Closest in time.
Gupta, T., Kamath, A., Kembhavi, A., Hoiem, D.: Towards General Purpose Vision Systems. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16399–16409 (2022)
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…