2018

Zero-Shot Detection

Zhu, Pengkai, Wang, Hanxiao, Saligrama, Venkatesh

Understand

As we move towards large-scale object detection, it is unrealistic to expect annotated training data, in the form of bounding box annotations around objects, for all object classes at sufficient scale, and so methods capable of unseen object detection are required.

We propose a novel zero-shot method based on training an end-to-end model that fuses semantic attribute prediction with visual features to propose object bounding boxes for seen and unseen classes.
While we utilize semantic features during training, our method is agnostic to semantic information for unseen classes at test-time.
Our method retains the efficiency and effectiveness of YOLOv2 for objects seen during training, while improving its performance for novel and unseen objects.

Built on

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in IEEE Conference on Computer Vision and Pattern Recognition . IEEE, 2009, pp. 248–255
2009
Earlier work this paper cites.
A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, “Describing objects by their attributes,” in IEEE Conference on Computer Vision and Pattern Recognition . IEEE, 2009, pp. 1778–1785
2009
Earlier work this paper cites.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International Journal of Computer Vision , vol. 88, no. 2, pp. 303–338, 2010
2010
Earlier work this paper cites.
D. Parikh and K. Grauman, “Interactively building a discriminative vocabulary of nameable attributes,” in Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on . IEEE, 2011, pp. 1681–1688
2011
Earlier work this paper cites.
T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka, “Metric learning for large scale image classification: Generalizing to new classes at near-zero cost,” Computer Vision–ECCV 2012 , pp. 488–501, 2012
2012
Earlier work this paper cites.
R. Socher, M. Ganjoo, C. D. Manning, and A. Ng, “Zero-shot learning through cross-modal transfer,” in Advances in neural information processing systems , 2013, pp. 935–943
2013
Earlier work this paper cites.
A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, T. Mikolov et al. , “Devise: A deep visual-semantic embedding model,” in Advances in neural information processing systems , 2013, pp. 2121–2129
2013
Earlier work this paper cites.
J. R. Uijlings, K. E. Van De Sande, T. Gevers, and A. W. Smeulders, “Selective search for object recognition,” International journal of computer vision , vol. 104, no. 2, pp. 154–171, 2013
2013
Earlier work this paper cites.
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in neural information processing systems , 2013, pp. 3111–3119
2013
Earlier work this paper cites.
C. H. Lampert, H. Nickisch, and S. Harmeling, “Attribute-based classification for zero-shot visual object categorization,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 36, no. 3, pp. 453–465, 2014
2014
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European Conference on Computer Vision . Springer, 2014, pp. 740–755
2014
Earlier work this paper cites.
C. L. Zitnick and P. Dollár, “Edge boxes: Locating object proposals from edges,” in European Conference on Computer Vision . Springer, 2014, pp. 391–405
2014
Earlier work this paper cites.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Object detectors emerge in deep scene cnns,” arXiv preprint arXiv:1412.6856 , 2014
Original
2014
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
Original
2014
Earlier work this paper cites.
Z. Zhang and V. Saligrama, “Zero-shot learning via semantic similarity embedding,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 4166–4174
2015
Earlier work this paper cites.
R. Girshick, “Fast r-cnn,” in IEEE International Conference on Computer Vision , 2015, pp. 1440–1448
2015
Earlier work this paper cites.
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing Systems , 2015, pp. 91–99
2015
Earlier work this paper cites.

Similar

J. Lei Ba, K. Swersky, S. Fidler et al. , “Predicting deep zero-shot convolutional neural networks using textual descriptions,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 4247–4255
2015
Cited alongside, same era.
J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 3431–3440
2015
Cited alongside, same era.
Z. Zhang, Y. Liu, T. Bolukbasi, M.-M. Cheng, and V. Saligrama, “Bing++: A fast high quality object proposal generator at 100fps,” arXiv preprint arXiv:1511.04511 , 2015
Original
2015
Cited alongside, same era.
M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Is object localization for free?-weakly-supervised learning with convolutional neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 685–694
2015
Cited alongside, same era.
M. Cho, S. Kwak, C. Schmid, and J. Ponce, “Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 1201–1210
2015
Cited alongside, same era.
Z. Akata, S. Reed, D. Walter, H. Lee, and B. Schiele, “Evaluation of output embeddings for fine-grained image classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 2927–2936
2015
Cited alongside, same era.
J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” arXiv preprint arXiv:1612.08242 , 2016
Original
2016
Cited alongside, same era.
Z. Zhang and V. Saligrama, “Zero-shot learning via joint latent similarity embedding,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016
2016
Cited alongside, same era.
——, “Zero-shot learning via joint latent similarity embedding,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 6034–6042
2016
Cited alongside, same era.
——, “Zero-shot recognition via structured prediction,” in European conference on computer vision . Springer, 2016, pp. 533–548
2016
Cited alongside, same era.
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 779–788
2016
Cited alongside, same era.
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in European Conference on Computer Vision . Springer, 2016, pp. 21–37
2016
Cited alongside, same era.

Then

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Later among the works it cites.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 2818–2826
2016
Later among the works it cites.
Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, “Label-embedding for image classification,” IEEE transactions on pattern analysis and machine intelligence , vol. 38, no. 7, pp. 1425–1438, 2016
2016
Later among the works it cites.
G. Patterson and J. Hays, “Coco attributes: Attributes for people, animals, and objects,” in European Conference on Computer Vision . Springer, 2016, pp. 85–100
2016
Later among the works it cites.
W.-L. Chao, S. Changpinyo, B. Gong, and F. Sha, “An empirical study and analysis of generalized zero-shot learning for object recognition in the wild,” in ECCV , 2016
2016
Later among the works it cites.
——, “Learning for new visual environments with limited labels,” arXiv preprint arXiv:1901.09079 , 2019
Original
2016
Later among the works it cites.
Y. Fu, T. Xiang, Y.-G. Jiang, X. Xue, L. Sigal, and S. Gong, “Recent advances in zero-shot recognition,” arXiv preprint arXiv:1710.04837 , 2017
Original
2017
Later among the works it cites.
Y. Xian, B. Schiele, and Z. Akata, “Zero-shot learning-the good, the bad and the ugly,” in 30th IEEE Conference on Computer Vision and Pattern Recognition , 2017
2017
Later among the works it cites.
K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” IEEE International Conference on Computer Vision , 2017
2017
Later among the works it cites.
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” arXiv preprint arXiv:1708.02002 , 2017
Original
2017
Later among the works it cites.
E. Park, W. Liu, O. Russakovsky, J. Deng, L. Fei-Fei, and A. Berg, “Ilsvrc-2017,” URL http://www. image-net. org/challenges/LSVRC/2017 , 2017
2017
Later among the works it cites.
Y. Chen, J. Wang, Y. Bai, G. Castañón, and V. Saligrama, “Probabilistic semantic retrieval for surveillance videos with activity graphs,” IEEE Transactions on Multimedia , 2018
2018
Closest in time.
A. Bansal, K. Sikka, G. Sharma, R. Chellappa, and A. Divakaran, “Zero-shot object detection,” arXiv preprint arXiv:1804.04340 , 2018
Original
2018
Closest in time.
S. Rahman, S. Khan, and F. Porikli, “Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts,” arXiv preprint arXiv:1803.06049 , 2018
Original
2018
Closest in time.
V. Kumar Verma, G. Arora, A. Mishra, and P. Rai, “Generalized zero-shot learning via synthesized examples,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018
2018
Closest in time.
Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly,” IEEE transactions on pattern analysis and machine intelligence , 2018
2018
Closest in time.
P. Zhu, H. Wang, and V. Saligrama, “Generalized zero-shot recognition based on visually semantic embedding,” arXiv preprint arXiv:1811.07993 , 2018
Original
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…