2021

Polarized Self-Attention: Towards High-quality Pixel-wise Regression

Liu, Huajun, Liu, Fuqiang, Fan, Xinyi et al.

Understand

Pixel-wise regression is probably the most common problem in fine-grained computer vision tasks, such as estimating keypoint heatmaps and segmentation masks.

These regression problems are very challenging particularly because they require, at low computation overheads, modeling long-range dependencies on high-resolution inputs/outputs to estimate the highly nonlinear pixel-wise semantics.
While attention mechanisms in Deep Convolutional Neural Networks(DCNNs) has become popular for boosting long-range dependencies, element-specific attention, such as Nonlocal blocks, is highly complex and noise-sensitive to learn, and most of simplified attention hybrids try to reach the best compromise among multiple types of tasks.
In this paper, we present the Polarized Self-Attention(PSA) block that incorporates two critical designs towards high-quality pixel-wise regression: (1) Polarized filtering: keeping high internal resolution in both channel and spatial attention computation while completely collapsing input tensors along their counterpart dimensions.

Built on

Microsoft coco: Common objects in context, 2014
Original
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár · 2014
Earlier work this paper cites.
The pascal visual object classes challenge: A retrospective
Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman · 2015
Earlier work this paper cites.
Fast r-cnn
Ross Girshick · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
Olga Russakovsky, Hao Su Jia Deng, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei · 2015
Earlier work this paper cites.
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele · 2016
Earlier work this paper cites.
Deep Learning
Ian Goodfellow, Yoshua Bengio, and Aaron Courville · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Efficient piecewise training of deep structured models for semantic segmentation
Guosheng Lin, Chunhua Shen, Anton van dan Hengel, and Ian Reid · 2016
Earlier work this paper cites.
Stacked hourglass networks for human pose estimation
Alejandro Newell, Kaiyu Yang, and Jia Deng · 2016
Earlier work this paper cites.
Speeding up semantic segmentation for autonomous driving
Michael Treml, José Arjona-Medina, Thomas Unterthiner, Rupesh Durgesh, Felix Friedmann, Peter Schuberth, Andreas Mayr, Martin Heusel, Markus Hofmarcher, Michael Widrich, Bernhard Nessler, and Sepp Hochreiter · 2016
Earlier work this paper cites.
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille · 2017
Earlier work this paper cites.
Residual conv-deconv grid network for semantic segmentation
Damien Fourure, Rémi Emonet, Elisa Fromont, Damien Muselet, Alain Tremeau, and Christian Wolf · 2017
Earlier work this paper cites.
Mask r-cnn
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick · 2017
Earlier work this paper cites.
Video scene parsing with predictive feature learning
Xiaojie Jin, Xin Li, Huaxin Xiao, Xiaohui Shen, Zhe Lin, Jimei Yang, Yunpeng Chen, Jian Dong, Luoqi Liu, Zequn Jie, Jiashi Feng, and Shuicheng Yan · 2017
Earlier work this paper cites.
Refinenet: Multi-path refinement networks for high-resolution semantic segmentation
Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid · 2017
Earlier work this paper cites.
A survey on deep learning in medical image analysis
Geert Litjens, Thijs Kooi, Babak Ehteshami, Bejnordi Arnaud, Arindra Adiyoso, and Setio Francesco · 2017
Earlier work this paper cites.
Large kernel matters – improve semantic segmentation by global convolutional network
Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun · 2017
Earlier work this paper cites.
Full-resolution residual networks for semantic segmentation in street scenes
Tobias Pohlen, Alexander Hermans, Markus Mathias, and Bastian Leibe · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit abd Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Scale-adaptive convolutions for scene parsing
Rui Zhang, Sheng Tang, Yongdong Zhang, Jintao Li, and Shuicheng Yan · 2017
Earlier work this paper cites.
Pyramid scene parsing network
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia · 2017
Earlier work this paper cites.
Cascaded pyramid network for multi-person pose estimation
Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun · 2018
Earlier work this paper cites.

Similar

Augmented reality driving using semantic geo-registration
Han-Pang Chiu, Varun Murali, Ryan Villamil, G. Drew Kessler, Supun Samarasekera, and Rakesh Kumar · 2018
Cited alongside, same era.
Gather-excite: Exploiting feature context in convolutional neural networks
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi · 2018
Cited alongside, same era.
Squeeze-and-excitation networks
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu · 2018
Cited alongside, same era.
Recurrent scene parsing with perspective understanding in the loop
Shu Kong and Charless Fowlkes · 2018
Cited alongside, same era.
Dynamic-structured semantic propagation network
Xiaodan Liang, Hongfei Zhou, and Eric Xing · 2018
Cited alongside, same era.
Bam: bottleneck attention module
Jongchan Park, Sanghyun Woo, Joon-Young Lee, and In SoKweon · 2018
Cited alongside, same era.
Self-attention with relative position representations
Original
Ashish Vaswani Peter Shaw, Jakob Uszkoreit · 2018
Cited alongside, same era.
Understanding convolution for semantic segmentation
Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cottrell · 2018
Cited alongside, same era.
Non-local neural networks
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He · 2018
Cited alongside, same era.
Cbam: Convolutional block attention module
Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon · 2018
Cited alongside, same era.
Simple baselines for human pose estimation and tracking
Bin Xiao, Haiping Wu, and Yichen Wei · 2018
Cited alongside, same era.
Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing
Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe · 2018
Cited alongside, same era.

Then

Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation
Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei-Fei · 2019
Later among the works it cites.
Stand-alone self-attention in vision models
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens · 2019
Later among the works it cites.
Deep high-resolution representation learning for human pose estimation
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang · 2019
Later among the works it cites.
Wider or deeper: Revisiting the resnet model for visual recognition
Zifeng Wu, Chunhua Shen, and Anton van den Hengel · 2019
Later among the works it cites.
Second-order non-local attention networks for person re-identification
Bryan (Ning) Xia, Yuan Gong, Yizhe Zhang, and Christian Poellabauer · 2019
Later among the works it cites.
Co-occurrent features in semantic segmentation
Hang Zhang, Chenguang Wang, and Junyuan Xie · 2019
Later among the works it cites.
Residual non-local attention networks for image restoration
Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu · 2019
Later among the works it cites.
Objects as points
Original
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl · 2019
Later among the works it cites.
Deformable convnets v2: More deformable, better results
Original
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai · 2019
Later among the works it cites.
Asymmetric non-local neural networks for semantic segmentation
Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai · 2019
Later among the works it cites.
Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation
Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen · 2020
Later among the works it cites.
On the relationship between self-attention and convolutional layers
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi · 2020
Later among the works it cites.
The devil is in the details: Delving into unbiased data processing for human pose estimation
Junjie Huang, Zheng Zhu, Feng Guo, and Guan Huang · 2020
Later among the works it cites.
Efficient attention: Attention with linear complexities, 2020
Original
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li · 2020
Later among the works it cites.
Hierarchical multi-scale attention for semantic segmentation
Original
Andrew Tao, Karan Sapra, and Bryan Catanzaro · 2020
Later among the works it cites.
Deep high-resolution representation learning for visual recognition
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, , and Bin Xiao · 2020
Later among the works it cites.
Object-contextual representations for semantic segmentation
Yuhui Yuan, Xilin Chen, and Jingdong Wang · 2020
Later among the works it cites.
Distribution-aware coordinate representation for human pose estimation
Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu · 2020
Later among the works it cites.
Squeeze-and-attention networks for semantic segmentation
Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Zhifeng Li, Wei-Shi Zheng, Jonathan Li, and Alexander Wong · 2020
Later among the works it cites.
Rethinking the heatmap regression for bottom-up human pose estimation
Zhengxiong Luo, Zhicheng Wang, Yan Huang, Tieniu Tan, and Erjin Zhou · 2021
Closest in time.
Pixel-wise crowd understanding via synthetic data
Qi Wang, Junyu Gao, Wei Lin, and Yuan Yuan · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…