2023

All in Tokens: Unifying Output Space of Visual Tasks via Soft Token

Ning, Jia, Li, Chen, Zhang, Zheng et al.

Understand

Unlike language tasks, where the output space is usually limited to a set of tokens, the output space of visual tasks is more complicated, making it difficult to build a unified visual model for various visual tasks.

In this paper, we seek to unify the output space of visual tasks, so that we can also build a unified model for visual tasks.
To this end, we demonstrate a single unified model that simultaneously handles two typical visual tasks of instance segmentation and depth estimation, which have discrete/fixed-length and continuous/varied-length outputs, respectively.
We propose several new techniques that take into account the particularity of visual tasks: 1) Soft token.

Built on

Indoor segmentation and support inference from rgbd images
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus · 2012
Earlier work this paper cites.
Depth map prediction from a single image using a multi-scale deep network
David Eigen, Christian Puhrsch, and Rob Fergus · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick · 2014
Earlier work this paper cites.
Deep convolutional neural fields for depth estimation from a single image
Fayao Liu, Chunhua Shen, and Guosheng Lin · 2015
Earlier work this paper cites.
Learning ordinal relationships for mid-level vision
Daniel Zoran, Phillip Isola, Dilip Krishnan, and William T. Freeman · 2015
Earlier work this paper cites.
Single-image depth perception in the wild
Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng · 2016
Earlier work this paper cites.
Mask r-cnn
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick · 2017
Earlier work this paper cites.
Neural discrete representation learning
Aaron Van Den Oord, Oriol Vinyals, et al · 2017
Earlier work this paper cites.
Deep ordinal regression network for monocular depth estimation
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao · 2018
Earlier work this paper cites.
Geonet: Geometric neural network for joint depth and surface normal estimation
Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia · 2018
Earlier work this paper cites.
Structured attention guided convolutional neural fields for monocular depth estimation
Dan Xu, Wei Wang, Hao Tang, Hong Liu, Nicu Sebe, and Elisa Ricci · 2018
Earlier work this paper cites.
Soft labels for ordinal regression
Raul Diaz and Amit Marathe · 2019
Earlier work this paper cites.
Monocular depth estimation using relative depth maps
Jaehan Lee and Chang-Su Kim · 2019
Earlier work this paper cites.
From big to small: Multi-scale local planar guidance for monocular depth estimation
Original
Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh · 2019
Earlier work this paper cites.
Real-time joint semantic segmentation and depth estimation using asymmetric annotations
Vladimir Nekrasov, Thanuja Dharmasiri, Andrew Spek, Tom Drummond, Chunhua Shen, and Ian D. Reid · 2019
Earlier work this paper cites.
Enforcing geometric constraints of virtual normal for depth prediction
Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan · 2019
Earlier work this paper cites.
Pattern-affinitive propagation across depth, surface normal and semantic segmentation
Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang · 2019
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Earlier work this paper cites.

Similar

End-to-end object detection with transformers
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko · 2020
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al · 2020
Cited alongside, same era.
Sdc-depth: Semantic divide-and-conquer network for monocular depth estimation
Lijun Wang, Jianming Zhang, Oliver Wang, Zhe Lin, and Huchuan Lu · 2020
Cited alongside, same era.
Polarmask: Single shot instance segmentation with polar representation
Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo · 2020
Cited alongside, same era.
Dense reppoints: Representing visual objects with dense point sets
Ze Yang, Yinghao Xu, Han Xue, Zheng Zhang, Raquel Urtasun, Liwei Wang, Stephen Lin, and Han Hu · 2020
Cited alongside, same era.
Beit: Bert pre-training of image transformers
Original
Hangbo Bao, Li Dong, and Furu Wei · 2021
Cited alongside, same era.
Adabins: Depth estimation using adaptive bins
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka · 2021
Cited alongside, same era.
Pix2seq: A language modeling framework for object detection
Original
Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton · 2021
Cited alongside, same era.
Taming transformers for high-resolution image synthesis
Patrick Esser, Robin Rombach, and Bjorn Ommer · 2021
Cited alongside, same era.
Cutdepth: Edge-aware data augmentation in depth estimation
Original
Yasunori Ishii and Takayoshi Yamashita · 2021
Cited alongside, same era.
Perceiver io: A general architecture for structured inputs & outputs
Original
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Cited alongside, same era.

Then

Masked-attention mask transformer for universal image segmentation
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar · 2022
Later among the works it cites.
Vector quantized diffusion model for text-to-image synthesis
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo · 2022
Later among the works it cites.
Global-local path networks for monocular depth estimation with vertical cutdepth
Original
Doyeon Kim, Woonghyun Ga, Pyunghwan Ahn, Donggyu Joo, Sewhan Chun, and Junmo Kim · 2022
Later among the works it cites.
Uvim: A unified modeling approach for vision with learned guiding codes
Original
Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby · 2022
Later among the works it cites.
Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation
Original
Zhenyu Li, Zehui Chen, Xianming Liu, and Junjun Jiang · 2022
Later among the works it cites.
Binsformer: Revisiting adaptive bins for monocular depth estimation
Original
Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang · 2022
Later among the works it cites.
Swin transformer v2: Scaling up capacity and resolution
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al · 2022
Later among the works it cites.
Unified-io: A unified model for vision, language, and multi-modal tasks
Original
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi · 2022
Later among the works it cites.
P3depth: Monocular depth estimation with a piecewise planarity prior
Vaishakh Patil, Christos Sakaridis, Alexander Liniger, and Luc Van Gool · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Later among the works it cites.
Image as a foreign language: Beit pretraining for all vision and vision-language tasks
Original
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al · 2022
Later among the works it cites.
Revealing the dark secrets of masked image modeling
Original
Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao · 2022
Later among the works it cites.
Simmim: A simple framework for masked image modeling
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu · 2022
Later among the works it cites.
Inverted pyramid multi-task transformer for dense scene understanding
Hanrong Ye and Dan Xu · 2022
Later among the works it cites.
Coca: Contrastive captioners are image-text foundation models
Original
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu · 2022
Later among the works it cites.
New crfs: Neural window fully-connected crfs for monocular depth estimation
Original
Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan · 2022
Later among the works it cites.
Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks
Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…