2023

Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection

Wang, Shihao, Liu, Yingfei, Wang, Tiancai et al.

Understand

In this paper, we propose a long-sequence modeling framework, named StreamPETR, for multi-view 3D object detection.

Built upon the sparse query design in the PETR series, we systematically develop an object-centric temporal mechanism.
The model is performed in an online manner and the long-term historical information is propagated through object queries frame by frame.
Besides, we introduce a motion-aware layer normalization to model the movement of the objects.

Built on

Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick · 2014
Earlier work this paper cites.
A learned representation for artistic style
Original
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Decoupled weight decay regularization
Original
Ilya Loshchilov and Frank Hutter · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Video object detection with locally-weighted deformable neighbors
Zhengkai Jiang, Peng Gao, Chaoxu Guo, Qian Zhang, Shiming Xiang, and Chunhong Pan · 2019
Earlier work this paper cites.
An energy and gpu-computation efficient backbone network for real-time object detection
Youngwan Lee, Joong-won Hwang, Sangrok Lee, Yuseok Bae, and Jongyoul Park · 2019
Earlier work this paper cites.
Gaugan: semantic image synthesis with spatially adaptive normalization
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu · 2019
Earlier work this paper cites.
Objects365: A large-scale, high-quality dataset for object detection
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun · 2019
Earlier work this paper cites.
Class-balanced grouping and sampling for point cloud 3d object detection
Original
Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu · 2019
Earlier work this paper cites.
nuscenes: A multimodal dataset for autonomous driving
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom · 2020
Earlier work this paper cites.
End-to-end object detection with transformers
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko · 2020
Earlier work this paper cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al · 2020
Earlier work this paper cites.
Scalability in perception for autonomous driving: Waymo open dataset
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al · 2020
Earlier work this paper cites.
Deformable detr: Deformable transformers for end-to-end object detection
Original
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai · 2020
Earlier work this paper cites.
Bevdet4d: Exploit temporal cues in multi-camera 3d object detection
Junjie Huang and Guan Huang · 2021
Earlier work this paper cites.
Bevdet: High-performance multi-camera 3d object detection in bird-eye-view
Original
Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du · 2021
Earlier work this paper cites.
Swin transformer: Hierarchical vision transformer using shifted windows
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo · 2021
Earlier work this paper cites.
Is pseudo-lidar needed for monocular 3d object detection?
Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon · 2021
Earlier work this paper cites.

Similar

Fcos3d: Fully convolutional one-stage monocular 3d object detection
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin · 2021
Cited alongside, same era.
3d-man: 3d multi-frame attention network for object detection
Zetong Yang, Yin Zhou, Zhifeng Chen, and Jiquan Ngiam · 2021
Cited alongside, same era.
Center-based 3d object detection and tracking
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl · 2021
Cited alongside, same era.
Memot: multi-object tracking with memory
Jiarui Cai, Mingze Xu, Wei Li, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto · 2022
Cited alongside, same era.
Reversible column networks
Original
Yuxuan Cai, Yizhuang Zhou, Qi Han, Jianjian Sun, Xiangwen Kong, Jun Li, and Xiangyu Zhang · 2022
Cited alongside, same era.
Polar parametrization for vision-based surround-view 3d detection
Original
Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Chang Huang, and Wenyu Liu · 2022
Cited alongside, same era.
Flashattention: Fast and memory-efficient exact attention with io-awareness
Original
Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré · 2022
Cited alongside, same era.
Aedet: Azimuth-invariant multi-view 3d object detection
Original
Chengjian Feng, Zequn Jie, Yujie Zhong, Xiangxiang Chu, and Lin Ma · 2022
Cited alongside, same era.
Cc-3dt: Panoramic 3d object tracking via cross-camera fusion
Original
Tobias Fischer, Yung-Hsu Yang, Suryansh Kumar, Min Sun, and Fisher Yu · 2022
Cited alongside, same era.
Queryprop: Object query propagation for high-performance video object detection
Fei He, Naiyu Gao, Jian Jia, Xin Zhao, and Kaiqi Huang · 2022
Cited alongside, same era.
Monocular quasi-dense 3d object tracking
Hou-Ning Hu, Yung-Hsu Yang, Tobias Fischer, Trevor Darrell, Fisher Yu, and Min Sun · 2022
Cited alongside, same era.
Bevpoolv2: A cutting-edge implementation of bevdet toward deployment
Original
Junjie Huang and Guan Huang · 2022
Cited alongside, same era.

Then

Petrv2: A unified framework for 3d perception from multi-camera images
Original
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun · 2022
Later among the works it cites.
Detr4d: Direct multi-view 3d object detection with sparse attention
Original
Zhipeng Luo, Changqing Zhou, Gongjie Zhang, and Shijian Lu · 2022
Later among the works it cites.
Trackformer: Multi-object tracking with transformers
Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer · 2022
Later among the works it cites.
Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection
Original
Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, and Wei Zhan · 2022
Later among the works it cites.
Focal-petr: Embracing foreground for efficient multi-camera 3d object detection
Original
Shihao Wang, Xiaohui Jiang, and Ying Li · 2022
Later among the works it cites.
Mv-fcos3d++: Multi-view camera-only 4d object detection with pretrained monocular backbones
Original
Tai Wang, Qing Lian, Chenming Zhu, Xinge Zhu, and Wenwei Zhang · 2022
Later among the works it cites.
Detr3d: 3d object detection from multi-view images via 3d-to-2d queries
Yue Wang, Guizilini Vitor Campagnolo, Tianyuan Zhang, Hang Zhao, and Justin Solomon · 2022
Later among the works it cites.
Bevformer v2: Adapting modern image backbones to bird’s-eye-view recognition via perspective supervision
Original
Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al · 2022
Later among the works it cites.
Quality matters: Embracing quality clues for robust 3d multi-object tracking
Original
Jinrong Yang, En Yu, Zeming Li, Xiaoping Li, and Wenbing Tao · 2022
Later among the works it cites.
Motr: End-to-end multiple-object tracking with transformer
Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei · 2022
Later among the works it cites.
Mutr3d: A multi-camera tracking framework via 3d-to-2d queries
Tianyuan Zhang, Xuanyao Chen, Yue Wang, Yilun Wang, and Hang Zhao · 2022
Later among the works it cites.
Motrv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors
Original
Yuang Zhang, Tiancai Wang, and Xiangyu Zhang · 2022
Later among the works it cites.
Fast-bev: Towards real-time on-vehicle bird’s-eye view perception
Original
Bin Huang, Yangguang Li, Enze Xie, Feng Liang, Luya Wang, Mingzhu Shen, Fenggang Liu, Tianqi Wang, Ping Luo, and Jing Shao · 2023
Closest in time.
Bevformer ++ : Improving bevformer for 3d camera-only object detection: 1st place solution for waymo open dataset challenge 2022
Zhiqi Li, Hanming Deng, Tianyu Li, Yangyi Huang, Chonghao Sima, Xiangwei Geng, Yulu Gao, Wenhai Wang, Yang Li, and Lewei Lu · 2023
Closest in time.
Standing between past and future: Spatio-temporal modeling for multi-camera 3d multi-object tracking
Original
Ziqi Pang, Jie Li, Pavel Tokmakov, Dian Chen, Sergey Zagoruyko, and Yu-Xiong Wang · 2023
Closest in time.
Simpletrack: Understanding and rethinking 3d multi-object tracking
Ziqi Pang, Zhichao Li, and Naiyan Wang · 2023
Closest in time.
Object as query: Equipping any 2d object detector with 3d detection ability
Original
Zitian Wang, Zehao Huang, Jiahui Fu, Naiyan Wang, and Si Liu · 2023
Closest in time.
Adding conditional control to text-to-image diffusion models
Original
Lvmin Zhang and Maneesh Agrawala · 2023
Closest in time.
Bytetrackv2: 2d and 3d multi-object tracking by associating every detection box
Original
Yifu Zhang, Xinggang Wang, Xiaoqing Ye, Wei Zhang, Jincheng Lu, Xiao Tan, Errui Ding, Peize Sun, and Jingdong Wang · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…