2021

P2T: Pyramid Pooling Transformer for Scene Understanding

Wu, Yu-Huan, Liu, Yun, Zhan, Xin et al.

Understand

Recently, the vision transformer has achieved great success by pushing the state-of-the-art of various vision tasks.

One of the most challenging problems in the vision transformer is that the large sequence length of image tokens leads to high computational cost (quadratic complexity).
A popular solution to this problem is to use a single pooling operation to reduce the sequence length.
This paper considers how to improve existing vision transformers, where the pooled feature extracted by a single pooling operation seems less powerful.

Built on

K. Grauman and T. Darrell, “The pyramid match kernel: Discriminative classification with sets of image features,” in Int. Conf. Comput. Vis. , vol. 2. IEEE, 2005, pp. 1458–1465
2005
Earlier work this paper cites.
S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories,” in IEEE Conf. Comput. Vis. Pattern Recog. , vol. 2. IEEE, 2006, pp. 2169–2178
2006
Earlier work this paper cites.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” Int. J. Comput. Vis. , vol. 88, no. 2, pp. 303–338, 2010
2010
Earlier work this paper cites.
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Int. Conf. Artif. Intell. Stat. , 2010, pp. 249–256
2010
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Adv. Neural Inform. Process. Syst. , 2012, pp. 1097–1105
2012
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in Eur. Conf. Comput. Vis. , 2014, pp. 740–755
2014
Earlier work this paper cites.
O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand et al. , “Findings of the 2014 workshop on statistical machine translation,” in The Workshop on Statistical Machine Translation , 2014, pp. 12–58
2014
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Int. Conf. Learn. Represent. , 2015
2015
Earlier work this paper cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2015, pp. 1–9
2015
Earlier work this paper cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , “ImageNet large scale visual recognition challenge,” Int. J. Comput. Vis. , vol. 115, no. 3, pp. 211–252, 2015
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 37, no. 9, pp. 1904–1916, 2015
2015
Earlier work this paper cites.
D. Yoo, S. Park, J.-Y. Lee, and I. So Kweon, “Multi-scale pyramid pooling for deep convolutional representation,” in Int. Conf. Comput. Vis. Worksh. , 2015, pp. 71–80
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2016, pp. 770–778
2016
Earlier work this paper cites.
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2016, pp. 3213–3223
2016
Earlier work this paper cites.
J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450 , 2016
Original
2016
Earlier work this paper cites.
D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs),” arXiv preprint arXiv:1606.08415 , 2016
Original
2016
Earlier work this paper cites.
B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ADE20K dataset,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2017, pp. 633–641
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Adv. Neural Inform. Process. Syst. , 2017, pp. 6000–6010
2017
Earlier work this paper cites.
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2017, pp. 1492–1500
2017
Earlier work this paper cites.
H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2017, pp. 2881–2890
2017
Earlier work this paper cites.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “MobileNets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861 , 2017
Original
2017
Earlier work this paper cites.
W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi, “A survey of deep neural network architectures and their applications,” Neurocomputing , vol. 234, pp. 11–26, 2017
2017
Earlier work this paper cites.
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 40, no. 4, pp. 834–848, 2017
2017
Earlier work this paper cites.
P. Zhang, D. Wang, H. Lu, H. Wang, and X. Ruan, “Amulet: Aggregating multi-level convolutional features for salient object detection,” in Int. Conf. Comput. Vis. , 2017, pp. 202–211
2017
Earlier work this paper cites.
T. Wang, A. Borji, L. Zhang, P. Zhang, and H. Lu, “A stagewise refinement model for detecting salient objects in images,” in Int. Conf. Comput. Vis. , 2017, pp. 4019–4028
2017
Earlier work this paper cites.
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in Int. Conf. Comput. Vis. , 2017, pp. 2980–2988
2017
Earlier work this paper cites.
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Int. Conf. Learn. Represent. , 2017
2017
Earlier work this paper cites.

Similar

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted residuals and linear bottlenecks,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2018, pp. 4510–4520
2018
Cited alongside, same era.
N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, “ShuffleNet v2: Practical guidelines for efficient CNN architecture design,” in Eur. Conf. Comput. Vis. , 2018, pp. 116–131
2018
Cited alongside, same era.
X. Zhang, X. Zhou, M. Lin, and J. Sun, “ShuffleNet: An extremely efficient convolutional neural network for mobile devices,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2018, pp. 6848–6856
2018
Cited alongside, same era.
M. M. K. Sarker, H. A. Rashwan, F. Akram, S. F. Banu, A. Saleh, V. K. Singh, F. U. Chowdhury, S. Abdulwahab, S. Romani, P. Radeva et al. , “Slsdeep: Skin lesion segmentation based on dilated residual and pyramid pooling networks,” in Med. Image. Comput. Comput. Assist. Interv. Springer, 2018, pp. 21–29
2018
Cited alongside, same era.
S.-W. Kim, H.-K. Kook, J.-Y. Sun, M.-C. Kang, and S.-J. Ko, “Parallel feature pyramid network for object detection,” in Eur. Conf. Comput. Vis. , 2018, pp. 234–250
2018
Cited alongside, same era.
J.-R. Chang and Y.-S. Chen, “Pyramid stereo matching network,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2018, pp. 5410–5418
2018
Cited alongside, same era.
D. Park, K. Kim, and S. Young Chun, “Efficient module based single image super resolution for multiple problems,” in IEEE Conf. Comput. Vis. Pattern Recog. Worksh. , 2018, pp. 882–890
2018
Cited alongside, same era.
H. Zhang and V. M. Patel, “Densely connected pyramid dehazing network,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2018, pp. 3194–3203
2018
Cited alongside, same era.
K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu et al. , “MMDetection: Open MMLab detection toolbox and benchmark,” arXiv preprint arXiv:1906.07155 , 2019
Original
2018
Cited alongside, same era.
G. Huang, Z. Liu, G. Pleiss, L. Van Der Maaten, and K. Weinberger, “Convolutional networks with dense connectivity,” IEEE Trans. Pattern Anal. Mach. Intell. , 2019
2019
Cited alongside, same era.
M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in Int. Conf. Mach. Learn. , 2019, pp. 6105–6114
2019
Cited alongside, same era.
A. Kirillov, R. Girshick, K. He, and P. Dollár, “Panoptic feature pyramid networks,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2019, pp. 6399–6408
2019
Cited alongside, same era.

Then

B. Heo, S. Yun, D. Han, S. Chun, J. Choe, and S. J. Oh, “Rethinking spatial dimensions of vision transformers,” in Int. Conf. Comput. Vis. , 2021, pp. 11 936–11 945
2021
Closest in time.
W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid Vision Transformer: A versatile backbone for dense prediction without convolutions,” in Int. Conf. Comput. Vis. , 2021
2021
Closest in time.
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical vision transformer using shifted windows,” in Int. Conf. Comput. Vis. , 2021, pp. 10 012–10 022
2021
Closest in time.
H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer, “Multiscale vision transformers,” in Int. Conf. Comput. Vis. , 2021, pp. 6824–6835
2021
Closest in time.
Y. Liu, Y.-H. Wu, G. Sun, L. Zhang, A. Chhatkuli, and L. Van Gool, “Vision transformers with hierarchical attention,” arXiv preprint arXiv:2106.03180 , 2021
Original
2021
Closest in time.
J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “SwinIR: Image restoration using Swin Transformer,” in Int. Conf. Comput. Vis. , 2021, pp. 1833–1844
2021
Closest in time.
X. Chu, Z. Tian, Y. Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen, “Twins: Revisiting the design of spatial attention in vision transformers,” in Adv. Neural Inform. Process. Syst. , 2021
2021
Closest in time.
W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “PVTv2: Improved baselines with pyramid vision transformer,” arXiv preprint arXiv:2106.13797 , 2021
Original
2021
Closest in time.
H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang, “CvT: Introducing convolutions to vision transformers,” in Int. Conf. Comput. Vis. , 2021, pp. 22–31
2021
Closest in time.
B. Graham, A. El-Nouby, H. Touvron, P. Stock, A. Joulin, H. Jégou, and M. Douze, “LeViT: a vision transformer in ConvNet’s clothing for faster inference,” in Int. Conf. Comput. Vis. , 2021, pp. 12 259–12 269
2021
Closest in time.
Q. Han, Z. Fan, Q. Dai, L. Sun, M.-M. Cheng, J. Liu, and J. Wang, “Demystifying local vision transformer: Sparse connectivity, weight sharing, and dynamic weight,” arXiv preprint arXiv:2106.04263 , 2021
Original
2021
Closest in time.
H. Wang, Y. Zhu, H. Adam, A. Yuille, and L.-C. Chen, “MaX-DeepLab: End-to-end panoptic segmentation with mask transformers,” in IEEE Conf. Comput. Vis. Pattern Recog. , 2021, pp. 5463–5474
2021
Closest in time.
R. Liu, Z. Yuan, T. Liu, and Z. Xiong, “End-to-end lane shape prediction with transformers,” in IEEE Winter Conf. Appl. Comput. Vis. , 2021, pp. 3694–3702
2021
Closest in time.
J. Hu, L. Cao, L. Yao, S. Zhang, Y. Wang, K. Li, F. Huang, R. Ji, and L. Shao, “ISTR: End-to-end instance segmentation with transformers,” arXiv preprint arXiv:2105.00637 , 2021
Original
2021
Closest in time.
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in Int. Conf. Mach. Learn. , 2021, pp. 10 347–10 357
2021
Closest in time.
L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, F. E. Tay, J. Feng, and S. Yan, “Tokens-to-token ViT: Training vision transformers from scratch on ImageNet,” in Int. Conf. Comput. Vis. , 2021, pp. 558–567
2021
Closest in time.
X. Chu, Z. Tian, B. Zhang, X. Wang, X. Wei, H. Xia, and C. Shen, “Conditional positional encodings for vision transformers,” arXiv preprint arXiv:2102.10882 , 2021
Original
2021
Closest in time.
Z. Jiang, Q. Hou, L. Yuan, D. Zhou, X. Jin, A. Wang, and J. Feng, “Token labeling: Training a 85.5% top-1 accuracy vision transformer with 56M parameters on ImageNet,” arXiv preprint arXiv:2104.10858 , 2021
Original
2021
Closest in time.
Y. Li, K. Zhang, J. Cao, R. Timofte, and L. Van Gool, “LocalViT: Bringing locality to vision transformers,” arXiv preprint arXiv:2104.05707 , 2021
Original
2021
Closest in time.
K. Yuan, S. Guo, Z. Liu, A. Zhou, F. Yu, and W. Wu, “Incorporating convolution designs into visual transformers,” in Int. Conf. Comput. Vis. , 2021, pp. 579–588
2021
Closest in time.
Y. Yuan, L. Huang, J. Guo, C. Zhang, X. Chen, and J. Wang, “Ocnet: Object context for semantic segmentation,” Int. J. Comput. Vis. , pp. 1–24, 2021
2021
Closest in time.
X. Lian, Y. Pang, J. Han, and J. Pan, “Cascaded hierarchical atrous spatial pyramid pooling module for semantic segmentation,” Pattern Recognition , vol. 110, p. 107622, 2021
2021
Closest in time.
Y.-H. Wu, Y. Liu, L. Zhang, W. Gao, and M.-M. Cheng, “Regularized densely-connected pyramid network for salient instance segmentation,” IEEE Trans. Image Process. , vol. 30, pp. 3897–3907, 2021
2021
Closest in time.
Y. Dong, J.-B. Cordonnier, and A. Loukas, “Attention is not all you need: Pure attention loses rank doubly exponentially with depth,” arXiv preprint arXiv:2103.03404 , 2021
Original
2021
Closest in time.
P. Zhang, X. Dai, J. Yang, B. Xiao, L. Yuan, L. Zhang, and J. Gao, “Multi-scale vision Longformer: A new vision transformer for high-resolution image encoding,” in Int. Conf. Comput. Vis. , 2021, pp. 2998–3008
2021
Closest in time.
J.-J. Liu, Q. Hou, Z.-A. Liu, and M.-M. Cheng, “Poolnet+: Exploring the potential of pooling for salient object detection,” IEEE TPAMI , 2022
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…