2019

Salient Object Detection in the Deep Learning Era: An In-Depth Survey

Wang, Wenguan, Lai, Qiuxia, Fu, Huazhu et al.

Understand

As an essential problem in computer vision, salient object detection (SOD) has attracted an increasing amount of research attention over the years.

Recent advances in SOD are predominantly led by deep learning-based solutions (named deep SOD).
To enable in-depth understanding of deep SOD, in this paper, we provide a comprehensive survey covering various aspects, ranging from algorithm taxonomy to unsolved issues.
In particular, we first review deep SOD algorithms from different perspectives, including network architecture, level of supervision, learning paradigm, and object-/instance-level detection.

Built on

E. L. Kaufman, M. W. Lord, T. W. Reese, and J. Volkmann, “The discrimination of visual number,”
1949
Earlier work this paper cites.
A. M. Treisman and G. Gelade, “A feature-integration theory of attention,”
1980
Earlier work this paper cites.
C. Koch and S. Ullman, “Shifts in selective visual attention: Towards the underlying neural circuitry,”
1985
Earlier work this paper cites.
R. Caruana, “Multitask learning,”
1997
Earlier work this paper cites.
L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual attention for rapid scene analysis,”
1998
Earlier work this paper cites.
D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in
2001
Earlier work this paper cites.
Y.-F. Ma, L. Lu, H.-J. Zhang, and M. Li, “A user attention model for video summarization,” in
2002
Earlier work this paper cites.
C. Bucilua, R. Caruana, and A. Niculescu-Mizil, “Model compression,” in
2006
Earlier work this paper cites.
S. Avidan and A. Shamir, “Seam carving for content-aware image resizing,” in
2007
Earlier work this paper cites.
T. Liu, J. Sun, N.-N. Zheng, X. Tang, and H.-Y. Shum, “Learning to detect a salient object,” in
2007
Earlier work this paper cites.
S. Alpert, M. Galun, R. Basri, and A. Brandt, “Image segmentation by probabilistic bottom-up aggregation and cue integration,” in
2007
Earlier work this paper cites.
D. Simakov, Y. Caspi, E. Shechtman, and M. Irani, “Summarizing visual data using bidirectional similarity,” in
2008
Earlier work this paper cites.
Z. Wang and B. Li, “A two-stage approach to saliency detection in images,” in
2008
Earlier work this paper cites.
R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, “Frequency-tuned salient region detection,” in
2009
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in
2009
Earlier work this paper cites.
C. Guo and L. Zhang, “A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression,”
2009
Earlier work this paper cites.
C. M. Masciocchi, S. Mihalas, D. Parkhurst, and E. Niebur, “Everyone knows what is interesting: Salient locations which should be fixated,”
2009
Earlier work this paper cites.
Y. Sugano, Y. Matsushita, and Y. Sato, “Calibration-free gaze sensing using saliency maps,” in
2010
Earlier work this paper cites.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,”
2010
Earlier work this paper cites.
V. Movahedi and J. H. Elder, “Design and perceptual validation of performance measures for salient object segmentation,” in
2010
Earlier work this paper cites.
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “Sun database: Large-scale scene recognition from abbey to zoo,” in
2010
Earlier work this paper cites.
M.-M. Cheng, G.-X. Zhang, N. J. Mitra, X. Huang, and S.-M. Hu, “Global contrast based salient region detection,”
2011
Earlier work this paper cites.
G. E. Hinton, A. Krizhevsky, and S. D. Wang, “Transforming auto-encoders,” in
2011
Earlier work this paper cites.
P. Krähenbühl and V. Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” in
2011
Earlier work this paper cites.
A. Torralba and A. A. Efros, “Unbiased look at dataset bias,” in
2011
Earlier work this paper cites.
F. Perazzi, P. Krähenbühl, Y. Pritch, and A. Hornung, “Saliency filters: Contrast based filtering for salient region detection,” in
2012
Earlier work this paper cites.
Y. Wei, F. Wen, W. Zhu, and J. Sun, “Geodesic saliency using background priors,”
2012
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in
2012
Earlier work this paper cites.
A. K. Mishra, Y. Aloimonos, L. F. Cheong, and A. Kassim, “Active visual segmentation,”
2012
Earlier work this paper cites.
R. Zhao, W. Ouyang, and X. Wang, “Unsupervised salience learning for person re-identification,” in
2013
Earlier work this paper cites.
J. Han, E. J. Pauwels, and P. De Zeeuw, “Fast saliency-aware multi-modality image fusion,”
2013
Earlier work this paper cites.
P. L. Rosin and Y.-K. Lai, “Artistic minimal rendering with lines and blocks,”
2013
Earlier work this paper cites.
Q. Yan, L. Xu, J. Shi, and J. Jia, “Hierarchical saliency detection,” in
2013
Earlier work this paper cites.
A. Borji and L. Itti, “State-of-the-art in visual attention modeling,”
2013
Earlier work this paper cites.
C. Yang, L. Zhang, H. Lu, X. Ruan, and M. H. Yang, “Saliency detection via graph-based manifold ranking,” in
2013
Earlier work this paper cites.
Z. Ren, S. Gao, L.-T. Chia, and I. W.-H. Tsang, “Region-based saliency detection and its application in object recognition.”
2014
Earlier work this paper cites.
S. Bi, G. Li, and Y. Yu, “Person re-identification using multiple experts with random subspaces,”
2014
Earlier work this paper cites.
A. Borji and L. Itti, “Defending yarbus: Eye movements reveal observers’ task,”
2014
Earlier work this paper cites.
S. Frintrop, G. M. García, and A. B. Cremers, “A cognitive approach for object discovery,” in
2014
Earlier work this paper cites.
W. Zhu, S. Liang, Y. Wei, and J. Sun, “Saliency optimization from robust background detection,” in
2014
Earlier work this paper cites.
Y. Li, X. Hou, C. Koch, J. M. Rehg, and A. L. Yuille, “The secrets of salient object segmentation,” in
2014
Earlier work this paper cites.
H. Peng, B. Li, W. Xiong, W. Hu, and R. Ji, “Rgbd salient object detection: a benchmark and algorithms,” in
2014
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in
2014
Earlier work this paper cites.
J. L. Long, N. Zhang, and T. Darrell, “Do convnets learn correspondence?” in
2014
Earlier work this paper cites.
P. Arbeláez, J. Pont-Tuset, J. T. Barron, F. Marques, and J. Malik, “Multiscale combinatorial grouping,” in
2014
Earlier work this paper cites.
R. Margolin, L. Zelnik-Manor, and A. Tal, “How to evaluate foreground maps?” in
2014
Earlier work this paper cites.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in
2014
Earlier work this paper cites.
M. Jiang, J. Xu, and Q. Zhao, “Saliency in crowd,” in
2014
Earlier work this paper cites.
G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in
2014
Earlier work this paper cites.
A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,”
Original
2014
Earlier work this paper cites.
J.-Y. Zhu, J. Wu, Y. Xu, E. Chang, and Z. Tu, “Unsupervised object class discovery via saliency-guided multiple class learning,”
2015
Earlier work this paper cites.
F. Zhang, B. Du, and L. Zhang, “Saliency-guided unsupervised feature learning for scene classification,”
2015
Earlier work this paper cites.
H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt
2015
Earlier work this paper cites.
G. Li and Y. Yu, “Visual saliency based on multiscale deep features,” in
2015
Earlier work this paper cites.
L. Wang, H. Lu, X. Ruan, and M.-H. Yang, “Deep networks for saliency detection via local estimation and global search,” in
2015
Earlier work this paper cites.
R. Zhao, W. Ouyang, H. Li, and X. Wang, “Saliency detection by multi-context deep learning,” in
2015
Earlier work this paper cites.
A. Borji, M.-M. Cheng, H. Jiang, and J. Li, “Salient object detection: A benchmark,”
2015
Earlier work this paper cites.

Similar

J. Shi, Q. Yan, L. Xu, and J. Jia, “Hierarchical image saliency detection on extended cssd,”
2015
Cited alongside, same era.
S. He, R. W. Lau, W. Liu, Z. Huang, and Q. Yang, “Supercnn: A superpixelwise convolutional neural network for salient object detection,”
2015
Cited alongside, same era.
M.-M. Cheng, N. J. Mitra, X. Huang, P. H. S. Torr, and S.-M. Hu, “Global contrast based salient region detection,”
2015
Cited alongside, same era.
R. Ju, Y. Liu, T. Ren, L. Ge, and G. Wu, “Depth-aware salient object detection using anisotropic center-surround difference,”
2015
Cited alongside, same era.
M. Jiang, S. Huang, J. Duan, and Q. Zhao, “SALICON: Saliency in context,” in
2015
Cited alongside, same era.
J. Zhang, S. Ma, M. Sameki, S. Sclaroff, M. Betke, Z. Lin, X. Shen, B. Price, and R. Mech, “Salient object subitizing,” in
2015
Cited alongside, same era.
J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in
2015
Cited alongside, same era.
S. Xie and Z. Tu, “Holistically-nested edge detection,” in
2015
Cited alongside, same era.
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in
2015
Cited alongside, same era.
J. Zhang, S. Sclaroff, Z. Lin, X. Shen, B. Price, and R. Mech, “Minimum barrier salient object detection at 80 fps,” in
2015
Cited alongside, same era.
I. Mehmood, M. Sajjad, W. Ejaz, and S. W. Baik, “Saliency-directed prioritization of visual data in wireless surveillance networks,”
2015
Cited alongside, same era.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in
2015
Cited alongside, same era.

Then

H. Song, W. Wang, S. Zhao, J. Shen, and K.-M. Lam, “Pyramid dilated deeper convlstm for video salient object detection,” in
2018
Later among the works it cites.
X. Wang, S. You, X. Li, and H. Ma, “Weakly-supervised semantic segmentation by iteratively mining common object features,” in
2018
Later among the works it cites.
W. Wang, J. Shen, and H. Ling, “A deep network solution for attention and aesthetics aware photo cropping,”
2018
Later among the works it cites.
N. Liu, J. Han, and M.-H. Yang, “Picanet: Learning pixel-wise contextual attention for saliency detection,” in
2018
Later among the works it cites.
T. V. Nguyen, Q. Zhao, and S. Yan, “Attentive systems: A survey,”
2018
Later among the works it cites.
D. Zhang, H. Fu, J. Han, A. Borji, and X. Li, “A review of co-saliency detection algorithms: fundamentals, applications, and challenges,”
2018
Later among the works it cites.
R. Cong, J. Lei, H. Fu, M.-M. Cheng, W. Lin, and Q. Huang, “Review of visual saliency detection with comprehensive information,”
2018
Later among the works it cites.
J. Han, D. Zhang, G. Cheng, N. Liu, and D. Xu, “Advanced deep-learning techniques for salient and category-specific object detection: a survey,”
2018
Later among the works it cites.
J. Zhang, T. Zhang, Y. Dai, M. Harandi, and R. Hartley, “Deep unsupervised saliency detection: A multiple noisy labeling perspective,” in
2018
Later among the works it cites.
C. Cao, Y. Hunag, Z. Wang, L. Wang, N. Xu, and T. Tan, “Lateral inhibition-inspired convolutional neural network for visual attention and saliency detection,” in
2018
Later among the works it cites.
X. Hu, L. Zhu, J. Qin, C.-W. Fu, and P.-A. Heng, “Recurrently aggregating deep features for salient object detection.” in
2018
Later among the works it cites.
M. Amirul Islam, M. Kalash, and N. D. B. Bruce, “Revisiting salient object detection: Simultaneous detection, ranking, and subitizing of multiple salient objects,” in
2018
Later among the works it cites.
L. Zhang, J. Dai, H. Lu, Y. He, and G. Wang, “A bi-directional message passing model for salient object detection,” in
2018
Later among the works it cites.
T. Wang, L. Zhang, S. Wang, H. Lu, G. Yang, X. Ruan, and A. Borji, “Detect globally, refine locally: A novel approach to saliency detection,” in
2018
Later among the works it cites.
X. Zhang, T. Wang, J. Qi, H. Lu, and G. Wang, “Progressive attention guided recurrent network for salient object detection,” in
2018
Later among the works it cites.
W. Wang, J. Shen, X. Dong, and A. Borji, “Salient object detection driven by fixation prediction,” in
2018
Later among the works it cites.
S. Chen, X. Tan, B. Wang, and X. Hu, “Reverse attention for salient object detection,” in
2018
Later among the works it cites.
G. Li, Y. Xie, and L. Lin, “Weakly supervised salient object detection using image labels,” in
2018
Later among the works it cites.
X. Li, F. Yang, H. Cheng, W. Liu, and D. Shen, “Contour knowledge transfer for salient object detection,” in
2018
Later among the works it cites.
D.-P. Fan, M.-M. Cheng, J.-J. Liu, S.-H. Gao, Q. Hou, and A. Borji, “Salient objects in clutter: Bringing salient object detection to the foreground,” in
2018
Later among the works it cites.
R. Fan, Q. Hou, M.-M. Cheng, G. Yu, R. R. Martin, and S.-M. Hu, “Associating inter-image salient instances for weakly supervised semantic segmentation,” in
2018
Later among the works it cites.
D.-P. Fan, M.-M. Cheng, J.-J. Liu, S.-H. Gao, Q. Hou, and A. Borji, “Enhanced-alignment measure for binary foreground map evaluation,” in
2018
Later among the works it cites.
M. Berman, A. Rannen Triki, and M. B. Blaschko, “The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks,” in
2018
Later among the works it cites.
T.-W. Ke, J.-J. Hwang, Z. Liu, and S. X. Yu, “Adaptive affinity fields for semantic segmentation,” in
2018
Later among the works it cites.
A. Veit and S. Belongie, “Convolutional networks with adaptive inference graphs,” in
2018
Later among the works it cites.
A. Zlateski, R. Jaroensri, P. Sharma, and F. Durand, “On the importance of label quality for semantic segmentation,” in
2018
Later among the works it cites.
Q. Zheng, J. Jiao, Y. Cao, and R. W. Lau, “Task-driven webpage saliency,” in
2018
Later among the works it cites.
M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in
2018
Later among the works it cites.
Y. Liu, Q. Zhang, D. Zhang, and J. Han, “Employing deep part-object relationships for salient object detection,” in
2019
Closest in time.
Q. Qi, S. Zhao, J. Shen, and K.-M. Lam, “Multi-scale capsule attention-based salient object detection with multi-crossed layer connections,” in
2019
Closest in time.
A. Borji, “Saliency prediction in the deep learning era: Successes and limitations,”
2019
Closest in time.
A. Borji, M.-M. Cheng, Q. Hou, H. Jiang, and J. Li, “Salient object detection: A survey,”
2019
Closest in time.
B. Li, Z. Sun, and Y. Guo, “Supervae: Superpixelwise variational autoencoder for salient object detection,” in
2019
Closest in time.
Y. Zeng, P. Zhang, J. Zhang, Z. Lin, and H. Lu, “Towards high-resolution salient object detection,” in
2019
Closest in time.
Y. Zhuge, Y. Zeng, and H. Lu, “Deep embedding features for salient object detection,” in
2019
Closest in time.
Z. Wu, L. Su, and Q. Huang, “Cascaded partial decoder for fast and accurate salient object detection,” in
2019
Closest in time.
Y. Zeng, Y. Zhuge, H. Lu, L. Zhang, M. Qian, and Y. Yu, “Multi-source weak supervision for saliency detection,” in
2019
Closest in time.
J.-X. Zhao, J.-J. Liu, D.-P. Fan, Y. Cao, J. Yang, and M.-M. Cheng, “Egnet: Edge guidance network for salient object detection,” in
2019
Closest in time.
M. Feng, H. Lu, and E. Ding, “Attentive feedback network for boundary-aware salient object detection,” in
2019
Closest in time.
X. Qin, Z. Zhang, C. Huang, C. Gao, M. Dehghan, and M. Jagersand, “Basnet: Boundary-aware salient object detection,” in
2019
Closest in time.
R. Wu, M. Feng, W. Guan, D. Wang, H. Lu, and E. Ding, “A mutual learning method for salient object detection with intertwined multi-supervision,” in
2019
Closest in time.
W. Wang, S. Zhao, J. Shen, S. C. Hoi, and A. Borji, “Salient object detection with pyramid attention and salient edges,” in
2019
Closest in time.
J.-J. Liu, Q. Hou, M.-M. Cheng, J. Feng, and J. Jiang, “A simple pooling-based design for real-time salient object detection,” in
2019
Closest in time.
W. Wang, J. Shen, M.-M. Cheng, and L. Shao, “An iterative and cooperative top-down and bottom-up inference network for salient object detection,” in
2019
Closest in time.
Y. Xu, D. Xu, X. Hong, W. Ouyang, R. Ji, M. Xu, and G. Zhao, “Structured modeling of joint deep feature and prediction refinement for salient object detection,” in
2019
Closest in time.
L. Zhang, J. Zhang, Z. Lin, H. Lu, and Y. He, “Capsal: Leveraging captioning to boost semantics for salient object detection,” in
2019
Closest in time.
J. Su, J. Li, Y. Zhang, C. Xia, and Y. Tian, “Selectivity or invariance: Boundary-aware salient object detection,” in
2019
Closest in time.
Z. Wu, L. Su, and Q. Huang, “Stacked cross refinement network for edge-aware salient object detection,” in
2019
Closest in time.
Y. Zeng, Y. Zhuge, H. Lu, and L. Zhang, “Joint learning of saliency detection and weakly supervised semantic segmentation,” in
2019
Closest in time.
J. Zhao, J. Li, H. Liu, S. Yan, and J. Feng, “Fine-grained multi-human parsing,”
2019
Closest in time.
C. Li, R. Cong, J. Hou, S. Zhang, Y. Qian, and S. Kwong, “Nested network with two-stream pyramid for salient object detection in optical remote sensing images,”
2019
Closest in time.
G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, “Learning efficient object detection models with knowledge distillation,” in
2019
Closest in time.
G. Sun, W. Wang, J. Dai, and L. Van Gool, “Mining cross-image semantics for weakly supervised semantic segmentation,” in
2020
Closest in time.
L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,”
2020
Closest in time.
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in
2057
Closest in time.
H. Jiang, J. Wang, Z. Yuan, Y. Wu, N. Zheng, and S. Li, “Salient object detection: A discriminative regional feature integration approach,” in
2090
Closest in time.
A. Karpathy, S. Miller, and L. Fei-Fei, “Object discovery in 3d scenes via shape analysis,” in
2095
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…