2020

UP-DETR: Unsupervised Pre-training for Object Detection with Transformers

Dai, Zhigang, Cai, Bolun, Lin, Yugeng et al.

Understand

DEtection TRansformer (DETR) for object detection reaches competitive performance compared with Faster R-CNN via a transformer encoder-decoder architecture.

However, trained with scratch transformers, DETR needs large-scale training data and an extreme long training schedule even on COCO dataset.
Inspired by the great success of pre-training transformers in natural language processing, we propose a novel pretext task named random query patch detection in Unsupervised Pre-training DETR (UP-DETR).
Specifically, we randomly crop patches from the given image and then feed them as queries to the decoder.

Built on

R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) , vol. 2. IEEE, 2006, pp. 1735–1742
2006
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition . Ieee, 2009, pp. 248–255
2009
Earlier work this paper cites.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International journal of computer vision , vol. 88, no. 2, pp. 303–338, 2010
2010
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European Conference on Computer Vision . Springer, 2014, pp. 740–755
2014
Earlier work this paper cites.
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing Systems , 2015, pp. 91–99
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in European Conference on Computer Vision . Springer, 2016, pp. 21–37
2016
Earlier work this paper cites.
R. Stewart, M. Andriluka, and A. Y. Ng, “End-to-end people detection in crowded scenes,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2016, pp. 2325–2333
2016
Earlier work this paper cites.
L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in European conference on computer vision . Springer, 2016, pp. 850–865
2016
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , 2017, pp. 5998–6008
2017
Earlier work this paper cites.
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in Proceedings of IEEE International Conference on Computer Vision , 2017, pp. 2980–2988
2017
Earlier work this paper cites.
Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei, “Fully convolutional instance-aware semantic segmentation,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2017, pp. 2359–2367
2017
Earlier work this paper cites.
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2017, pp. 2117–2125
2017
Earlier work this paper cites.
K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in Proceedings of IEEE International Conference on Computer Vision , 2017, pp. 2961–2969
2017
Earlier work this paper cites.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in Proceedings of European Conference on Computer Vision , 2018, pp. 132–149
2018
Earlier work this paper cites.
Z. Cai and N. Vasconcelos, “Cascade R-CNN: Delving into high quality object detection,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018, pp. 6154–6162
2018
Earlier work this paper cites.

Similar

H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints,” in Proceedings of European Conference on Computer Vision , 2018, pp. 734–750
2018
Cited alongside, same era.
Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018, pp. 3733–3742
2018
Cited alongside, same era.
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations , 2018
2018
Cited alongside, same era.
B. Li, J. Yan, W. Wu, Z. Zhu, and X. Hu, “High performance visual tracking with siamese region proposal network,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018, pp. 8971–8980
2018
Cited alongside, same era.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” OpenAI Blog , vol. 1, no. 8, p. 9, 2019
2019
Cited alongside, same era.
Y. Asano, C. Rupprecht, and A. Vedaldi, “Self-labelling via simultaneous clustering and representation learning,” in International Conference on Learning Representations , 2019
2019
Cited alongside, same era.
Z. Tian, C. Shen, H. Chen, and T. He, “Fcos: Fully convolutional one-stage object detection,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 9627–9636
2019
Cited alongside, same era.
X. Zhou, D. Wang, and P. Krähenbühl, “Objects as points,” arXiv preprint arXiv:1904.07850 , 2019
Original
2019
Cited alongside, same era.
H. Zhang and J. Wang, “Towards adversarially robust object detection,” in Proceedings of IEEE International Conference on Computer Vision , 2019, pp. 421–430
2019
Cited alongside, same era.
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” in Advances in Neural Information Processing Systems , 2019, pp. 5753–5763
2019
Cited alongside, same era.
M. Ye, X. Zhang, P. C. Yuen, and S.-F. Chang, “Unsupervised embedding learning via invariant and spreading instance feature,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 6210–6219
2019
Cited alongside, same era.
L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon, “Unified language model pre-training for natural language understanding and generation,” in Advances in Neural Information Processing Systems , 2019, pp. 13 063–13 075
2019
Cited alongside, same era.

Then

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” arXiv preprint arXiv:2005.12872 , 2020
Original
2020
Closest in time.
S. Zhang, C. Chi, Y. Yao, Z. Lei, and S. Z. Li, “Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 9759–9768
2020
Closest in time.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 9729–9738
2020
Closest in time.
X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” arXiv preprint arXiv:2003.04297 , 2020
Original
2020
Closest in time.
M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” Advances in Neural Information Processing Systems , vol. 33, 2020
2020
Closest in time.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” arXiv preprint arXiv:2002.05709 , 2020
Original
2020
Closest in time.
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al. , “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in Neural Information Processing Systems , vol. 33, 2020
2020
Closest in time.
Y. Cao, Z. Xie, B. Liu, Y. Lin, Z. Zhang, and H. Hu, “Parametric instance classification for unsupervised visual feature learning,” Advances in Neural Information Processing Systems , vol. 33, 2020
2020
Closest in time.
Y. Wu, Y. Chen, L. Yuan, Z. Liu, L. Wang, H. Li, and Y. Fu, “Rethinking classification and localization for object detection,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 10 186–10 195
2020
Closest in time.
G. Song, Y. Liu, and X. Wang, “Revisiting the sibling head in object detector,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 11 563–11 572
2020
Closest in time.
X. Li, L. Zhang, Y. P. Chen, Y.-W. Tai, and C.-K. Tang, “One-shot object detection without fine-tuning,” arXiv preprint arXiv:2005.03819 , 2020
Original
2020
Closest in time.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” In International Conference on Learning Representations , 2021
2021
Closest in time.
L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, F. E. H. Tay, J. Feng, and S. Yan, “Tokens-to-token vit: Training vision transformers from scratch on imagenet,” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 538–547, 2021
2021
Closest in time.
K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, “Transformer in transformer,” Advances in Neural Information Processing Systems , vol. 34, pp. 15 908–15 919, 2021
2021
Closest in time.
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 9992–10 002, 2021
2021
Closest in time.
K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 000–16 009
2022
Closest in time.
A. Bar, X. Wang, V. Kantorov, C. J. Reed, R. Herzig, G. Chechik, A. Rohrbach, T. Darrell, and A. Globerson, “Detreg: Unsupervised pretraining with region priors for object detection,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 14 605–14 615
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…