2022

CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers

Xu, Runsheng, Tu, Zhengzhong, Xiang, Hao et al.

Understand

Bird's eye view (BEV) semantic segmentation plays a crucial role in spatial sensing for autonomous driving.

Although recent literature has made significant progress on BEV map understanding, they are all based on single-agent camera-based systems.
These solutions sometimes have difficulty handling occlusions or detecting distant objects in complex traffic scenes.
Vehicle-to-Vehicle (V2V) communication technologies have enabled autonomous vehicles to share sensing information, dramatically improving the perception performance and range compared to single-agent systems.

Built on

Inverse perspective mapping simplifies optical flow computation and obstacle detection
H. A. Mallot, H. H. Bülthoff, J. Little, and S. Bohrer · 1991
Earlier work this paper cites.
Stacked convolutional auto-encoders for hierarchical feature extraction
J. Masci, U. Meier, D. Cireşan, and J. Schmidhuber · 2011
Earlier work this paper cites.
Spatial transformer networks
M. Jaderberg, K. Simonyan, A. Zisserman, et al · 2015
Earlier work this paper cites.
Layer normalization
Original
J. L. Ba, J. R. Kiros, and G. E. Hinton · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Earlier work this paper cites.
Sgdr: Stochastic gradient descent with warm restarts
Original
I. Loshchilov and F. Hutter · 2016
Earlier work this paper cites.
Identity mappings in deep residual networks
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Earlier work this paper cites.
Attention is all you need
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin · 2017
Earlier work this paper cites.
CARLA: An open urban driving simulator
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun · 2017
Earlier work this paper cites.
Decoupled weight decay regularization
Original
I. Loshchilov and F. Hutter · 2017
Earlier work this paper cites.
Collaborative automated driving: A machine learning-based method to enhance the accuracy of shared information
Z. Y. Rawashdeh and Z. Wang · 2018
Earlier work this paper cites.
Orthographic feature transform for monocular 3d object detection
Original
T. Roddick, A. Kendall, and R. Cipolla · 2018
Earlier work this paper cites.
A geometric approach to obtain a bird’s eye view from an image
S. Ammar Abbas and A. Zisserman · 2019
Earlier work this paper cites.
Deep learning based vehicle position and orientation estimation via inverse perspective mapping image
Y. Kim and D. Kum · 2019
Earlier work this paper cites.
Cooper: Cooperative perception for connected autonomous vehicles based on 3d point clouds
Q. Chen, S. Tang, Q. Yang, and S. Fu · 2019
Earlier work this paper cites.
F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds
Q. Chen, X. Ma, S. Tang, J. Guo, Q. Yang, and S. Fu · 2019
Earlier work this paper cites.
The right (angled) perspective: Improving the understanding of road scenes using boosted inverse perspective mapping
T. Bruls, H. Porav, L. Kunze, and P. Newman · 2019
Earlier work this paper cites.
Efficientnet: Rethinking model scaling for convolutional neural networks
M. Tan and Q. Le · 2019
Earlier work this paper cites.
Pointpillars: Fast encoders for object detection from point clouds
A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom · 2019
Earlier work this paper cites.
BEV-Seg: Bird’s eye view semantic segmentation using geometry and semantic point cloud
Original
M. H. Ng, K. Radia, J. Chen, D. Wang, I. Gog, and J. E. Gonzalez · 2020
Earlier work this paper cites.

Similar

Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d
J. Philion and S. Fidler · 2020
Cited alongside, same era.
nuscenes: A multimodal dataset for autonomous driving
H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom · 2020
Cited alongside, same era.
V2vnet: Vehicle-to-vehicle communication for joint perception and prediction
T.-H. Wang, S. Manivasagam, M. Liang, B. Yang, W. Zeng, and R. Urtasun · 2020
Cited alongside, same era.
A sim2real deep learning approach for the transformation of images from multiple vehicle-mounted cameras to a semantically segmented image in bird’s eye view
L. Reiher, B. Lampe, and L. Eckstein · 2020
Cited alongside, same era.
Cross-view semantic segmentation for sensing surroundings
B. Pan, J. Sun, H. Y. T. Leung, A. Andonian, and B. Zhou · 2020
Cited alongside, same era.
Deformable detr: Deformable transformers for end-to-end object detection
Original
X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai · 2020
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al · 2020
Cited alongside, same era.
Axial-deeplab: Stand-alone axial-attention for panoptic segmentation
H. Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, and L.-C. Chen · 2020
Cited alongside, same era.
Monolayout: Amodal scene layout from a single image
K. Mani, S. Daga, S. Garg, S. S. Narasimhan, M. Krishna, and K. M. Jatavallabhula · 2020
Cited alongside, same era.
Fiery: Future instance prediction in bird’s-eye view from surround monocular cameras
A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V. Badrinarayanan, R. Cipolla, and A. Kendall · 2021
Cited alongside, same era.
Projecting your view attentively: Monocular road scene layout estimation via cross-view transformation
W. Yang, Q. Li, W. Liu, Y. Yu, Y. Ma, S. He, and J. Pan · 2021
Cited alongside, same era.
Translating images into maps
Original
A. Saha, O. M. Maldonado, C. Russell, and R. Bowden · 2021
Cited alongside, same era.

Then

Improved multiscale vision transformers for classification and detection
Original
Y. Li, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong, J. Malik, and C. Feichtenhofer · 2021
Later among the works it cites.
Pyramid vision transformer: A versatile backbone for dense prediction without convolutions
W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao · 2021
Later among the works it cites.
Multiscale vision transformers
H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer · 2021
Later among the works it cites.
Vitae: Vision transformer advanced by exploring intrinsic inductive bias
Y. Xu, Q. Zhang, J. Zhang, and D. Tao · 2021
Later among the works it cites.
Xcit: Cross-covariance image transformers
A. Ali, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek, et al · 2021
Later among the works it cites.
Pay attention to mlps
H. Liu, Z. Dai, D. So, and Q. Le · 2021
Later among the works it cites.
Fnet: Mixing tokens with fourier transforms
Original
J. Lee-Thorp, J. Ainslie, I. Eckstein, and S. Ontanon · 2021
Later among the works it cites.
Shuffle transformer: Rethinking spatial shuffle for vision transformer
Original
Z. Huang, Y. Ben, G. Luo, P. Cheng, G. Yu, and B. Fu · 2021
Later among the works it cites.
Opencda: an open cooperative driving automation framework integrated with co-simulation
R. Xu, Y. Guo, X. Han, X. Xia, H. Xiang, and J. Ma · 2021
Later among the works it cites.
Efficientnetv2: Smaller models and faster training
M. Tan and Q. Le · 2021
Later among the works it cites.
Cross-view transformers for real-time map-view semantic segmentation
Original
B. Zhou and P. Krähenbühl · 2022
Closest in time.
Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers
Original
Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and J. Dai · 2022
Closest in time.
V2x-vit: Vehicle-to-everything cooperative perception with vision transformer
Original
R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, and J. Ma · 2022
Closest in time.
Keypoints-based deep feature fusion for cooperative vehicle detection of autonomous driving
Y. Yuan, H. Cheng, and M. Sester · 2022
Closest in time.
Maxim: Multi-axis mlp for image processing
Original
Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li · 2022
Closest in time.
Peripheral vision transformer
Original
J. Min, Y. Zhao, C. Luo, and M. Cho · 2022
Closest in time.
Global context vision transformers
Original
A. Hatamizadeh, H. Yin, J. Kautz, and P. Molchanov · 2022
Closest in time.
Maxvit: Multi-axis vision transformer
Original
Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li · 2022
Closest in time.
Model-agnostic multi-agent perception framework
Original
W. Chen, R. Xu, H. Xiang, L. Liu, and J. Ma · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…