2022

Peripheral Vision Transformer

Min, Juhong, Zhao, Yucheng, Luo, Chong et al.

Understand

Human vision possesses a special type of visual processing systems called peripheral vision.

Partitioning the entire visual field into multiple contour regions based on the distance to the center of our gaze, the peripheral vision provides us the ability to perceive various visual features at different regions.
In this work, we take a biologically inspired approach and explore to model peripheral vision in deep neural networks for visual recognition.
We propose to incorporate peripheral position encoding to the multi-head self-attention layers to let the network learn to partition the visual field into diverse peripheral regions given training data.

Built on

On seeing sidelong
J. Lettvin · 1976
Earlier work this paper cites.
Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition
K. Fukushima and S. Miyake · 1982
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner · 1998
Earlier work this paper cites.
Peripheral-foveal vision for real-time object recognition and tracking in video
S. Gould, J. Arfvidsson, A. Kaehler, B. Sapp, M. Messner, G. Bradski, P. Baumstarck, S. Chung, and A. Y. Ng · 2007
Earlier work this paper cites.
A summary-statistic representation in peripheral vision explains visual crowding
B. Balas, L. Nakano, and R. Rosenholtz · 2009
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
The contributions of central versus peripheral vision to scene gist recognition
A. M. Larson and L. C. Loschky · 2009
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
V. Nair and G. E. Hinton · 2010
Earlier work this paper cites.
Peripheral vision and pattern recognition: A review
H. Strasburger, I. Rentschler, and M. Jüttner · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Object recognition test in peripheral vision: A study on the influence of object color, pattern and shape
C. I. Lou, D. Migotina, J. P. Rodrigues, J. Semedo, F. Wan, P. U. Mak, P. I. Mak, M. I. Vai, F. Melicio, J. G. Pereira, and A. Rosa · 2012
Earlier work this paper cites.
Visualizing and understanding convolutional networks
M. D. Zeiler and R. Fergus · 2014
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
S. Ioffe and C. Szegedy · 2015
Earlier work this paper cites.
Layer normalization
Original
J. Ba, J. Kiros, and G. Hinton · 2016
Earlier work this paper cites.
Xception: Deep learning with depthwise separable convolutions
Original
F. Chollet · 2016
Earlier work this paper cites.
Can peripheral representations improve clutter metrics on complex scenes?
Original
A. Deza and M. P. Eckstein · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Earlier work this paper cites.
Capabilities and limitations of peripheral vision
R. Rosenholtz · 2016
Earlier work this paper cites.
Aggregated residual transformations for deep neural networks
Original
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He · 2016
Earlier work this paper cites.
Sideeye: A generative neural network based simulator of human peripheral vision
Original
L. Fridman, B. Jenik, S. Keshvari, B. Reimer, C. Zetzsche, and R. Rosenholtz · 2017
Earlier work this paper cites.
Revisiting unreasonable effectiveness of data in deep learning era
C. Sun, A. Shrivastava, S. Singh, and A. Gupta · 2017
Earlier work this paper cites.
Instance normalization: The missing ingredient for fast stylization
Original
D. Ulyanov, A. Vedaldi, and V. Lempitsky · 2017
Earlier work this paper cites.
Attention is all you need
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin · 2017
Earlier work this paper cites.
Central and peripheral vision for scene recognition: A neurocomputational modeling exploration
Original
P. Wang and G. W. Cottrell · 2017
Earlier work this paper cites.
Squeeze-and-excitation networks
J. Hu, L. Shen, and G. Sun · 2018
Earlier work this paper cites.
Neighbourhood consensus networks
I. Rocco, M. Cimpoi, R. Arandjelović, A. Torii, T. Pajdla, and J. Sivic · 2018
Earlier work this paper cites.
Self-attention with relative position representations
P. Shaw, J. Uszkoreit, and A. Vaswani · 2018
Earlier work this paper cites.
Context mitigates crowding: Peripheral object recognition in real-world images
M. W. Wijntjes and R. Rosenholtz · 2018
Earlier work this paper cites.

Similar

Transformer-xl: Attentive language models beyond a fixed-length context
Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov · 2019
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova · 2019
Cited alongside, same era.
Decoupled weight decay regularization
I. Loshchilov and F. Hutter · 2019
Cited alongside, same era.
Pytorch: An imperative style, high-performance deep learning library
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala · 2019
Cited alongside, same era.
Stand-alone self-attention in vision models
P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens · 2019
Cited alongside, same era.
End-to-end object detection with transformers
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko · 2020
Cited alongside, same era.
On the relationship between self-attention and convolutional layers
J.-B. Cordonnier, A. Loukas, and M. Jaggi · 2020
Cited alongside, same era.
Gaussian error linear units (gelus)
Original
D. Hendrycks and K. Gimpel · 2020
Cited alongside, same era.
Improve transformer models with better relative position embeddings
Z. Huang, D. Liang, P. Xu, and B. Xiang · 2020
Cited alongside, same era.
Demystifying visual awareness: Peripheral encoding plus limited decision complexity resolve the paradox of rich visual experience and curious perceptual failures
R. Rosenholtz · 2020
Cited alongside, same era.
Biologically inspired mechanisms for adversarial robustness
M. R. Vuyyuru, A. Banburski, N. Pant, and T. Poggio · 2020
Cited alongside, same era.
Axial-deeplab: Stand-alone axial-attention for panoptic segmentation
H. Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, and L.-C. Chen · 2020
Cited alongside, same era.

Then

Hypercorrelation squeeze for few-shot segmentation
J. Min, D. Kang, and M. Cho · 2021
Later among the works it cites.
Do vision transformers see like convolutional neural networks?
M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy · 2021
Later among the works it cites.
Bottleneck transformers for visual recognition
A. Srinivas, T.-Y. Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani · 2021
Later among the works it cites.
Segmenter: Transformer for semantic segmentation
R. Strudel, R. Garcia, I. Laptev, and C. Schmid · 2021
Later among the works it cites.
Mlp-mixer: An all-mlp architecture for vision
Original
I. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy · 2021
Later among the works it cites.
Resmlp: Feedforward networks for image classification with data-efficient training
Original
H. Touvron, P. Bojanowski, M. Caron, M. Cord, A. El-Nouby, E. Grave, G. Izacard, A. Joulin, G. Synnaeve, J. Verbeek, and H. Jégou · 2021
Later among the works it cites.
Training data-efficient image transformers & distillation through attention
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou · 2021
Later among the works it cites.
Augmenting convolutional networks with attention-based aggregation
Original
H. Touvron, M. Cord, A. El-Nouby, P. Bojanowski, A. Joulin, G. Synnaeve, J. Verbeek, and H. Jegou · 2021
Later among the works it cites.
Scaling local self-attention for parameter efficient visual backbones
A. Vaswani, P. Ramachandran, A. Srinivas, N. Parmar, B. Hechtman, and J. Shlens · 2021
Later among the works it cites.
Pyramid vision transformer: A versatile backbone for dense prediction without convolutions
W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao · 2021
Later among the works it cites.
Resnet strikes back: An improved training procedure in timm
Original
R. Wightman, H. Touvron, and H. Jégou · 2021
Later among the works it cites.
Cvt: Introducing convolutions to vision transformers
H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang · 2021
Later among the works it cites.
Rethinking and improving relative position encoding for vision transformer
K. Wu, H. Peng, M. Chen, J. Fu, and H. Chao · 2021
Later among the works it cites.
Early convolutions help transformers see better
T. Xiao, P. Dollar, M. Singh, E. Mintun, T. Darrell, and R. Girshick · 2021
Later among the works it cites.
Co-scale conv-attentional image transformers
W. Xu, Y. Xu, T. Chang, and Z. Tu · 2021
Later among the works it cites.
Focal self-attention for local-global interactions in vision transformers
J. Yang, C. Li, P. Zhang, X. Dai, B. Xiao, L. Yuan, and J. Gao · 2021
Later among the works it cites.
Metaformer is actually what you need for vision
Original
W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, J. Feng, and S. Yan · 2021
Later among the works it cites.
Tokens-to-token vit: Training vision transformers from scratch on imagenet
L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z.-H. Jiang, F. E. Tay, J. Feng, and S. Yan · 2021
Later among the works it cites.
A battle of network structures: An empirical study of cnn, transformer, and mlp
Original
Y. Zhao, G. Wang, C. Tang, C. Luo, W. Zeng, and Z.-J. Zha · 2021
Later among the works it cites.
Finding biological plausibility for adversarially robust features via metameric tasks
A. Harrington and A. Deza · 2022
Closest in time.
Pruning self-attentions into convolutional layers in single path
Original
H. He, J. Liu, Z. Pan, J. Cai, J. Zhang, D. Tao, and B. Zhuang · 2022
Closest in time.
Foveater: Foveated transformer for image classification
Original
A. Jonnalagadda, W. Y. Wang, B. S. Manjunath, and M. P. Eckstein · 2022
Closest in time.
Mpvit: Multi-path vision transformer for dense prediction
Y. Lee, J. Kim, J. Willette, and S. J. Hwang · 2022
Closest in time.
Uniformer: Unifying convolution and self-attention for visual recognition
Original
K. Li, Y. Wang, J. Zhang, P. Gao, G. Song, Y. Liu, H. Li, and Y. Qiao · 2022
Closest in time.
Nommer: Nominate synergistic context in vision transformer for visual recognition
H. Liu, X. Jiang, X. Li, Z. Bao, D. Jiang, and B. Ren · 2022
Closest in time.
A convnet for the 2020s
Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie · 2022
Closest in time.
MaxViT: Multi-Axis Vision Transformer
Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li · 2022
Closest in time.
Peripheral vision — Wikipedia, the free encyclopedia
Wikipedia contributors · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…