2022

SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation

Luo, Huaishao, Bao, Junwei, Wu, Youzheng et al.

Understand

Recently, the contrastive language-image pre-training, e.g., CLIP, has demonstrated promising results on various downstream tasks.

The pre-trained model can capture enriched visual concepts for images by learning from a large scale of text-image data.
However, transferring the learned visual knowledge to open-vocabulary semantic segmentation is still under-explored.
In this paper, we propose a CLIP-based model named SegCLIP for the topic of open-vocabulary segmentation in an annotation-free manner.

Built on

Efficient graph-based image segmentation
Felzenszwalb, P. F. and Huttenlocher, D. P · 2004
Earlier work this paper cites.
The pascal visual object classes (VOC) challenge
Everingham, M., Gool, L. V., Williams, C. K. I., Winn, J. M., and Zisserman, A · 2010
Earlier work this paper cites.
Microsoft COCO: common objects in context
Lin, T., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L · 2014
Earlier work this paper cites.
The role of context for object detection and semantic segmentation in the wild
Mottaghi, R., Chen, X., Liu, X., Cho, N., Lee, S., Fidler, S., Urtasun, R., and Yuille, A. L · 2014
Earlier work this paper cites.
Semantic image segmentation with deep convolutional nets and fully connected crfs
Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L · 2015
Earlier work this paper cites.
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., and Darrell, T · 2015
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., and Brox, T · 2015
Earlier work this paper cites.
The cityscapes dataset for semantic urban scene understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B · 2016
Earlier work this paper cites.
Gaussian error linear units (gelus)
Original
Hendrycks, D. and Gimpel, K · 2016
Earlier work this paper cites.
YFCC100M: the new data in multimedia research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L · 2016
Earlier work this paper cites.
Categorical reparameterization with gumbel-softmax
Jang, E., Gu, S., and Poole, B · 2017
Earlier work this paper cites.
Grad-cam: Visual explanations from deep networks via gradient-based localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017
Earlier work this paper cites.
Pyramid scene parsing network
Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J · 2017
Earlier work this paper cites.
Scene parsing through ade20k dataset
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A · 2017
Earlier work this paper cites.
Encoder-decoder with atrous separable convolution for semantic image segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H · 2018
Earlier work this paper cites.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning
Sharma, P., Ding, N., Goodman, S., and Soricut, R · 2018
Earlier work this paper cites.
Zero-shot semantic segmentation
Bucher, M., Vu, T., Cord, M., and Pérez, P · 2019
Earlier work this paper cites.
BERT: pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M., Lee, K., and Toutanova, K · 2019
Earlier work this paper cites.
Howto100m: Learning a text-video embedding by watching hundred million narrated video clips
Miech, A., Zhukov, D., Alayrac, J., Tapaswi, M., Laptev, I., and Sivic, J · 2019
Earlier work this paper cites.

Similar

Videobert: A joint model for video and language representation learning
Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C · 2019
Cited alongside, same era.
LXMERT: learning cross-modality encoder representations from transformers
Tan, H. and Bansal, M · 2019
Cited alongside, same era.
Semantic projection network for zero- and few-label semantic segmentation
Xian, Y., Choudhury, S., He, Y., Schiele, B., and Akata, Z · 2019
Cited alongside, same era.
UNITER: universal image-text representation learning
Chen, Y., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J · 2020
Cited alongside, same era.
Pixel-bert: Aligning image pixels with text by deep multi-modal transformers
Original
Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J · 2020
Cited alongside, same era.
Univl: A unified video and language pre-training model for multimodal understanding and generation
Original
Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Chen, X., and Zhou, M · 2020
Cited alongside, same era.
Phrasecut: Language-based image segmentation in the wild
Wu, C., Lin, Z., Cohen, S., Bui, T., and Maji, S · 2020
Cited alongside, same era.
Frozen in time: A joint video and image encoder for end-to-end retrieval
Bain, M., Nagrani, A., Varol, G., and Zisserman, A · 2021
Cited alongside, same era.
Emerging properties in self-supervised vision transformers
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A · 2021
Cited alongside, same era.
Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R · 2021
Cited alongside, same era.
Transformer interpretability beyond attention visualization
Chefer, H., Gur, S., and Wolf, L · 2021
Cited alongside, same era.
An empirical study of training self-supervised vision transformers
Chen, X., Xie, S., and He, K · 2021
Cited alongside, same era.

Then

LAION-400M: open dataset of clip-filtered 400 million image-text pairs
Original
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A · 2021
Later among the works it cites.
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H · 2021
Later among the works it cites.
Segformer: Simple and efficient design for semantic segmentation with transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., and Luo, P · 2021
Later among the works it cites.
Semantic segmentation in-the-wild without seeing any segmentation examples
Original
Zabari, N. and Hoshen, Y · 2021
Later among the works it cites.
Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H., and Zhang, L · 2021
Later among the works it cites.
Masked-attention mask transformer for universal image segmentation
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Girdhar, R · 2022
Closest in time.
Vision-language pre-training: Basics, recent advances, and future trends
Original
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., and Gao, J · 2022
Closest in time.
Open-vocabulary object detection via vision and language knowledge distillation
Gu, X., Lin, T.-Y., Kuo, W., and Cui, Y · 2022
Closest in time.
Masked autoencoders are scalable vision learners
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. B · 2022
Closest in time.
Oneformer: One transformer to rule universal image segmentation
Original
Jain, J., Li, J., Chiu, M., Hassani, A., Orlov, N., and Shi, H · 2022
Closest in time.
Open-vocabulary semantic segmentation with mask-adapted CLIP
Original
Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., and Marculescu, D · 2022
Closest in time.
Image segmentation using text and image prompts
Lüddecke, T. and Ecker, A. S · 2022
Closest in time.
Open-vocabulary semantic segmentation with frozen vision-language models
Original
Ma, C., Yang, Y., Wang, Y., Zhang, Y., and Xie, W · 2022
Closest in time.
Denseclip: Language-guided dense prediction with context-aware prompting
Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., and Lu, J · 2022
Closest in time.
Simvlm: Simple visual language model pretraining with weak supervision
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y · 2022
Closest in time.
Self-supervised visual representation learning with semantic grouping
Wen, X., Zhao, B., Zheng, A., Zhang, X., and Qi, X · 2022
Closest in time.
FILIP: fine-grained interactive language-image pre-training
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., and Xu, C · 2022
Closest in time.
Multi-grained vision language pre-training: Aligning texts with visual concepts
Zeng, Y., Zhang, X., and Li, H · 2022
Closest in time.
Regionclip: Region-based language-image pretraining
Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., and Gao, J · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…