2020

Learning to Discover Multi-Class Attentional Regions for Multi-Label Image Recognition

Gao, Bin-Bin, Zhou, Hong-Yu

Understand

Multi-label image recognition is a practical and challenging task compared to single-label image classification.

However, previous works may be suboptimal because of a great number of object proposals or complex attentional region generation modules.
In this paper, we propose a simple but efficient two-stream framework to recognize multi-category objects from global image to local regions, similar to how human beings perceive objects.
To bridge the gap between global and local streams, we propose a multi-class attentional region module which aims to make the number of attentional regions as small as possible and keep the diversity of these regions as high as possible.

Built on

D. Navon, “Forest before trees: The precedence of global features in visual perception,” Perception and Psychophysics , vol. 5, pp. 197–200, 1969
1969
Earlier work this paper cites.
S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , vol. 2, 2006, pp. 2169–2178
2006
Earlier work this paper cites.
J. Hegdé, “Time course of visual perception: coarse-to-fine processing and beyond,” Progress in Neurobiology , vol. 84, no. 4, pp. 405–439, 2008
2008
Earlier work this paper cites.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International Journal of Computer Vision (IJCV) , vol. 88, no. 2, pp. 303–338, 2010
2010
Earlier work this paper cites.
L. Bazzani, H. Larochelle, V. Murino, J.-a. Ting, and N. D. Freitas, “Learning attentional policies for tracking and recognition in video with deep networks,” in Proceedings of the 28th International Conference on Machine Learning (ICML) , 2011, pp. 937–944
2011
Earlier work this paper cites.
Z.-H. Zhou, M.-L. Zhang, S.-J. Huang, and Y.-F. Li, “Multi-instance multi-label learning,” Artificial Intelligence , vol. 176, no. 1, pp. 2291–2320, 2012
2012
Earlier work this paper cites.
M.-L. Zhang and Z.-H. Zhou, “A review on multi-label learning algorithms,” IEEE Transactions on Knowledge and Data Engineering (TKDE) , vol. 26, no. 8, pp. 1819–1837, 2013
2013
Earlier work this paper cites.
A. V. Flevaris, A. Martínez, and S. A. Hillyard, “Attending to global versus local stimulus features modulates neural processing of low versus high spatial frequencies: an analysis with event-related brain potentials,” Frontiers in psychology , vol. 5, 2014
2014
Earlier work this paper cites.
R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2014, pp. 580–587
2014
Earlier work this paper cites.
C. L. Zitnick and P. Dollár, “Edge boxes: Locating object proposals from edges,” in European Conference on Computer Vision (ECCV) , 2014, pp. 391–405
2014
Earlier work this paper cites.
M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P. Torr, “BING: Binarized normed gradients for objectness estimation at 300fps,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2014, pp. 3286–3293
2014
Earlier work this paper cites.
V. Mnih, N. Heess, A. Graves et al. , “Recurrent models of visual attention,” in Advances in Neural Information Processing Systems (NIPS) , 2014, pp. 2204–2212
2014
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in Advances in Neural Information Processing Systems (NIPS) , 2014, pp. 568–576
2014
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in European Conference on Computer Vision (ECCV) , 2014, pp. 740–755
2014
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2015, pp. 1026–1034
2015
Earlier work this paper cites.
J. Shao, K. Kang, C. Change Loy, and X. Wang, “Deeply learned attributes for crowded scene understanding,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2015, pp. 4657–4666
2015
Earlier work this paper cites.
Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2015, pp. 3730–3738
2015
Earlier work this paper cites.

Similar

Y. Wei, W. Xia, M. Lin, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan, “HCP: a flexible cnn framework for multi-label image classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , vol. 38, no. 9, pp. 1901–1907, 2015
2015
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , vol. 37, no. 9, pp. 1904–1916, 2015
2015
Cited alongside, same era.
Y. Li, C. Huang, C. C. Loy, and X. Tang, “Human attribute recognition by deep hierarchical contexts,” in European Conference on Computer Vision (ECCV) , 2016, pp. 684–700
2016
Cited alongside, same era.
H. Yang, J. Tianyi Zhou, Y. Zhang, B.-B. Gao, J. Wu, and J. Cai, “Exploit bounding box annotations for multi-label object recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 280–288
2016
Cited alongside, same era.
J. Wang, Y. Yang, J. Mao, Z. Huang, C. Huang, and W. Xu, “CNN-RNN: a unified framework for multi-label image classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 2285–2294
2016
Cited alongside, same era.
S. Hong, J. Oh, H. Lee, and B. Han, “Learning transferrable knowledge for semantic segmentation with deep convolutional neural network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 3204–3212
2016
Cited alongside, same era.
H. Lai, P. Yan, X. Shu, Y. Wei, and Y. Shuicheng, “Instance-aware hashing for multi-label image retrieval,” IEEE Transactions on Image Processing (TIP) , vol. 25, no. 6, pp. 2469–2479, 2016
2016
Cited alongside, same era.
X.-Y. Jing, F. Wu, Z. Li, R. Hu, and D. Zhang, “Multi-label dictionary learning for image annotation,” IEEE Transactions on Image Processing (TIP) , vol. 25, no. 6, pp. 2712–2725, 2016
2016
Cited alongside, same era.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 2921–2929
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 770–778
2016
Cited alongside, same era.
Z. Wang, T. Chen, G. Li, R. Xu, and L. Lin, “Multi-label image recognition by recurrently discovering attentional regions,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2017, pp. 464–472
2017
Cited alongside, same era.
F. Zhu, H. Li, W. Ouyang, N. Yu, and X. Wang, “Learning spatial regularization with image-level supervisions for multi-label image classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 5513–5522
2017
Cited alongside, same era.

Then

P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and visual question answering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 6077–6086
2018
Later among the works it cites.
C.-W. Lee, W. Fang, C.-K. Yeh, and Y.-C. Frank Wang, “Multi-label zero-shot learning with structured knowledge graphs,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 1576–1585
2018
Later among the works it cites.
W. Ge, S. Yang, and Y. Yu, “Multi-evidence filtering and fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 1277–1286
2018
Later among the works it cites.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 4510–4520
2018
Later among the works it cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations (ICLR) , 2015
Original
2018
Later among the works it cites.
H. Guo, K. Zheng, X. Fan, H. Yu, and S. Wang, “Visual attention consistency under image transforms for multi-label image classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019, pp. 729–739
2019
Later among the works it cites.
Z.-M. Chen, X.-S. Wei, P. Wang, and Y. Guo, “Multi-label image recognition with graph convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019, pp. 5177–5186
2019
Later among the works it cites.
T. Chen, M. Xu, X. Hui, H. Wu, and L. Lin, “Learning semantic-specific graph representation for multi-label image recognition,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2019, pp. 522–531
2019
Later among the works it cites.
W.-J. Yu, Z.-D. Chen, X. Luo, W. Liu, and X.-S. Xu, “DELTA: a deep dual-stream network for multi-label image classification,” Pattern Recognition (PR) , vol. 91, pp. 322–331, 2019
2019
Later among the works it cites.
J. Xu, H. Tian, Z. Wang, Y. Wang, F. Chen, and W. Kang, “Joint input and output space learning for multi-label image classification,” IEEE Transactions on Multimedia (TMM) , pp. 1–1, 2020
2020
Closest in time.
V. O. Yazici, A. Gonzalez-Garcia, A. Ramisa, B. Twardowski, and J. v. d. Weijer, “Orderless recurrent models for multi-label classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020, pp. 13 440–13 449
2020
Closest in time.
Z. Ji, B. Cui, H. Li, Y.-G. Jiang, T. Xiang, T. Hospedales, and Y. Fu, “Deep ranking for image zero-shot multi-label classification,” IEEE Transactions on Image Processing (TIP) , pp. 6549–6560, 2020
2020
Closest in time.
T. Chen, L. Lin, X. Hui, R. Chen, and H. Wu, “Knowledge-guided multi-label few-shot learning for general image recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , pp. 1–1, 2020
2020
Closest in time.
Z. Chen, X.-S. Wei, P. Wang, and Y. Guo, “Learning graph convolutional networks for multi-label recognition and applications,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2021
2021
Closest in time.
M. Jaderberg, K. Simonyan, A. Zisserman, and k. kavukcuoglu, “Spatial transformer networks,” in Advances in Neural Information Processing Systems (NIPS) , 2015, pp. 2017–2025
2025
Closest in time.
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in International Conference on Machine Learning (ICLR) , 2015, pp. 2048–2057
2057
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…