2023

SegGPT: Segmenting Everything In Context

Wang, Xinlong, Zhang, Xiaosong, Cao, Yue et al.

Understand

We present SegGPT, a generalist model for segmenting everything in context.

We unify various segmentation tasks into a generalist in-context learning framework that accommodates different kinds of segmentation data by transforming them into the same format of images.
The training of SegGPT is formulated as an in-context coloring problem with random color mapping for each data sample.
The objective is to accomplish diverse tasks according to the context, rather than relying on specific colors.

Built on

Adaptive background mixture models for real-time tracking
Chris Stauffer and W Eric L Grimson · 1999
Earlier work this paper cites.
Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response
AD Hoover, Valentina Kouznetsova, and Michael Goldbaum · 2000
Earlier work this paper cites.
Ridge-based vessel segmentation in color images of the retina
Joes Staal, Michael D Abràmoff, Meindert Niemeijer, Max A Viergever, and Bram Van Ginneken · 2004
Earlier work this paper cites.
The pascal visual object classes (voc) challenge
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman · 2009
Earlier work this paper cites.
An ensemble classification-based approach applied to retinal blood vessel segmentation
Muhammad Moazam Fraz, Paolo Remagnino, Andreas Hoppe, Bunyarit Uyyanonvara, Alicja R Rudnicka, Christopher G Owen, and Sarah A Barman · 2012
Earlier work this paper cites.
Robust vessel segmentation in fundus images
Attila Budai, Rüdiger Bock, Andreas Maier, Joachim Hornegger, and Georg Michelson · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Microsoft COCO: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick · 2014
Earlier work this paper cites.
Fast R-CNN
Ross Girshick · 2015
Earlier work this paper cites.
Fully convolutional networks for semantic segmentation
Jonathan Long, Evan Shelhamer, and Trevor Darrell · 2015
Earlier work this paper cites.
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele · 2016
Earlier work this paper cites.
Deep interactive object selection
Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas S Huang · 2016
Earlier work this paper cites.
Semantic instance segmentation with a discriminative loss function
Original
Bert De Brabandere, Davy Neven, and Luc Van Gool · 2017
Earlier work this paper cites.
Mask R-CNN
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick · 2017
Earlier work this paper cites.
Refinenet: Multi-path refinement networks for high-resolution semantic segmentation
Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid · 2017
Earlier work this paper cites.
The 2017 davis challenge on video object segmentation
Original
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Pyramid scene parsing network
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia · 2017
Earlier work this paper cites.
Look into person: Joint body parsing & pose estimation network and a new benchmark
Xiaodan Liang, Ke Gong, Xiaohui Shen, and Liang Lin · 2018
Earlier work this paper cites.

Similar

Iteratively trained interactive segmentation
Sabarinath Mahadevan, Paul Voigtlaender, and Bastian Leibe · 2018
Cited alongside, same era.
Youtube-vos: A large-scale video object segmentation benchmark
Original
Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang · 2018
Cited alongside, same era.
Semantic understanding of scenes through the ade20k dataset
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba · 2018
Cited alongside, same era.
Yolact: Real-time instance segmentation
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee · 2019
Cited alongside, same era.
A generative appearance model for end-to-end video object segmentation
Joakim Johnander, Martin Danelljan, Emil Brissman, Fahad Shahbaz Khan, and Michael Felsberg · 2019
Cited alongside, same era.
Panoptic feature pyramid networks
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár · 2019
Cited alongside, same era.
Agss-vos: Attention guided single-shot video object segmentation
Huaijia Lin, Xiaojuan Qi, and Jiaya Jia · 2019
Cited alongside, same era.
Video object segmentation using space-time memory networks
Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim · 2019
Cited alongside, same era.
isaid: A large-scale dataset for instance segmentation in aerial images
Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai · 2019
Cited alongside, same era.
Language models are few-shot learners
Original
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Cited alongside, same era.
End-to-end object detection with transformers
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko · 2020
Cited alongside, same era.
Video object segmentation with adaptive feature bank and uncertain-region refinement
Yongqing Liang, Xin Li, Navid Jafari, and Jim Chen · 2020
Cited alongside, same era.

Then

Loveda: A remote sensing land-cover dataset for domain adaptive semantic segmentation
Original
Junjue Wang, Zhuo Zheng, Ailong Ma, Xiaoyan Lu, and Yanfei Zhong · 2021
Later among the works it cites.
SOLO: A simple framework for instance segmentation
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li · 2021
Later among the works it cites.
Visual prompting via image inpainting
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei A. Efros · 2022
Later among the works it cites.
A unified sequence interface for vision tasks
Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J. Fleet, and Geoffrey Hinton · 2022
Later among the works it cites.
Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model
Ho Kei Cheng and Alexander G Schwing · 2022
Later among the works it cites.
Self-support few-shot semantic segmentation
Qi Fan, Wenjie Pei, Yu-Wing Tai, and Chi-Keung Tang · 2022
Later among the works it cites.
Cost aggregation with 4d convolutional swin transformer for few-shot segmentation
Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim · 2022
Later among the works it cites.
UViM: A unified modeling approach for vision with learned guiding codes
Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby · 2022
Later among the works it cites.
Recurrent dynamic embedding for video object segmentation
Mingxing Li, Li Hu, Zhiwei Xiong, Bang Zhang, Pan Pan, and Dong Liu · 2022
Later among the works it cites.
Swem: Towards real-time video object segmentation with sequential weighted expectation-maximization
Zhihui Lin, Tianyu Yang, Maomao Li, Ziyu Wang, Chun Yuan, Wenhao Jiang, and Wei Liu · 2022
Later among the works it cites.
Unified-IO: A unified model for vision, language, and multi-modal tasks
Original
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi · 2022
Later among the works it cites.
OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang · 2022
Later among the works it cites.
Doubly deformable aggregation of covariance matrices for few-shot segmentation
Zhitong Xiong, Haopeng Li, and Xiao Xiang Zhu · 2022
Later among the works it cites.
Feature-proxy transformer for few-shot segmentation
Jian-Wei Zhang, Yifan Sun, Yi Yang, and Wei Chen · 2022
Later among the works it cites.
Uni-Perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks
Original
Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai · 2022
Later among the works it cites.
Mose: A new dataset for video object segmentation in complex scenes
Original
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai · 2023
Closest in time.
Paco: Parts and attributes of common objects
Original
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al · 2023
Closest in time.
Images speak in images: A generalist painter for in-context visual learning
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…