2021

TrackFormer: Multi-Object Tracking with Transformers

Meinhardt, Tim, Kirillov, Alexander, Leal-Taixe, Laura et al.

Understand

The challenging task of multi-object tracking (MOT) requires simultaneous reasoning about track initialization, identity, and spatio-temporal trajectories.

We formulate this task as a frame-to-frame set prediction problem and introduce TrackFormer, an end-to-end trainable MOT approach based on an encoder-decoder Transformer architecture.
Our model achieves data association between frames via attention by evolving a set of track predictions through a video sequence.
The Transformer decoder initializes new tracks from static object queries and autoregressively follows existing tracks in space and time with the conceptually new and identity preserving track queries.

Built on

A linear programming approach for multiple object tracking
Hao Jiang, Sidney S. Fels, and James J. Little · 2007
Earlier work this paper cites.
Multiple target tracking using spatio-temporal markov chain monte carlo data association
Qian Yu, Gerard Medioni, and Isaac Cohen · 2007
Earlier work this paper cites.
Evaluating multiple object tracking performance: the clear mot metrics
Keni Bernardin and Rainer Stiefelhagen · 2008
Earlier work this paper cites.
Global data association for multi-object tracking using network flows
Li Zhang, Yuan Li, and Ramakant Nevatia · 2008
Earlier work this paper cites.
Object detection with discriminatively trained part based models
Pedro F. Felzenszwalb, Ross B. Girshick, David A. McAllester, and Deva Ramanan · 2009
Earlier work this paper cites.
You’ll never walk alone: modeling social behavior for multi-target tracking
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool · 2009
Earlier work this paper cites.
Learning pedestrian dynamics from the real world
Paul Scovanner and Marshall F. Tappen · 2009
Earlier work this paper cites.
Multiple target tracking in world coordinate with single, minimally calibrated camera
Wongun Choi and Silvio Savarese · 2010
Earlier work this paper cites.
Multi-target tracking by continuous energy minimization
Anton Andriyenko and Konrad Schindler · 2011
Earlier work this paper cites.
Multiple object tracking using k-shortest paths optimization
Jerome Berclaz, Francois Fleuret, Engin Turetken, and Pascal Fua · 2011
Earlier work this paper cites.
Everybody needs somebody: Modeling social and grouping behavior on a linear programming multiple people tracker
Laura Leal-Taixé, Gerard Pons-Moll, and Bodo Rosenhahn · 2011
Earlier work this paper cites.
Globally-optimal greedy algorithms for tracking a variable number of objects
Hamed Pirsiavash, Deva Ramanan, and Charless C. Fowlkes · 2011
Earlier work this paper cites.
Who are you with and where are you going?
Kota Yamaguchi, Alexander C. Berg, Luis E. Ortiz, and Tamara L. Berg · 2011
Earlier work this paper cites.
Learning an image-based motion context for multiple people tracking
Laura Leal-Taixé, Michele Fenzi, Alina Kuznetsova, Bodo Rosenhahn, and Silvio Savarese · 2014
Earlier work this paper cites.
Deepreid: Deep filter pairing neural network for person re-identification
Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
Original
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár · 2014
Earlier work this paper cites.
Multiple hypothesis tracking revisited
Chanho Kim, Fuxin Li, Arridhana Ciptadi, and James M. Rehg · 2015
Earlier work this paper cites.
Faster r-cnn: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun · 2015
Earlier work this paper cites.
Scalable person re-identification: A benchmark
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian · 2015
Earlier work this paper cites.
Social lstm: Human trajectory prediction in crowded spaces
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Learning by tracking: siamese cnn for robust target association
Laura Leal-Taixé, Cristian Canton-Ferrer, and Konrad Schindler · 2016
Earlier work this paper cites.
Mot16: A benchmark for multi-object tracking
Original
Anton Milan, Laura Leal-Taixé, Ian D. Reid, Stefan Roth, and Konrad Schindler · 2016
Earlier work this paper cites.
Performance measures and a data set for multi-target, multi-camera tracking
Ergys Ristani, Francesco Solera, Roger S. Zou, Rita Cucchiara, and Carlo Tomasi · 2016
Earlier work this paper cites.

Similar

Learning social etiquette: Human trajectory prediction
Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese · 2016
Cited alongside, same era.
End-to-end people detection in crowded scenes
Russell Stewart, Mykhaylo Andriluka, and Andrew Y Ng · 2016
Cited alongside, same era.
Exploit all the layers: Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers
Fan Yang, Wongun Choi, and Yuanqing Lin · 2016
Cited alongside, same era.
Exploit all the layers: Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers
Fan Yang, Wongun Choi, and Yuanqing Lin · 2016
Cited alongside, same era.
Online multi-object tracking using cnn-based single object tracker with spatial-temporal attention mechanism
Qi Chu, Wanli Ouyang, Hongsheng Li, Xiaogang Wang, Bin Liu, and Nenghai Yu · 2017
Cited alongside, same era.
Detect to track and track to detect
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman · 2017
Cited alongside, same era.
Mask r-cnn
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick · 2017
Cited alongside, same era.
Improvements to frank-wolfe optimization for multi-detector multi-object tracking
Roberto Henschel, Laura Leal-Taixé, Daniel Cremers, and Bodo Rosenhahn · 2017
Cited alongside, same era.
Multiple people tracking by lifted multicut and person re-identification
Siyu Tang, Mykhaylo Andriluka, Bjoern Andres, and Bernt Schiele · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Real-time multiple people tracking with deeply learned candidate selection and person re-identification
Long Chen, Haizhou Ai, Zijie Zhuang, and Chong Shang · 2018
Cited alongside, same era.
Learning to detect and track visible and occluded body joints in a virtual world
Matteo Fabbri, Fabio Lanzi, Simone Calderara, Andrea Palazzi, Roberto Vezzani, and Rita Cucchiara · 2018
Cited alongside, same era.

Then

End-to-end object detection with transformers
Nicolas Carion, F. Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko · 2020
Later among the works it cites.
Motchallenge: A benchmark for single-camera multiple target tracking
Patrick Dendorfer, Aljosa Osep, Anton Milan, Daniel Cremers, Ian Reid, Stefan Roth, and Laura Leal-Taixé · 2020
Later among the works it cites.
MOT20: A benchmark for multi object tracking in crowded scenes
Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian D. Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixé · 2020
Later among the works it cites.
Lifted disjoint paths with application in multiple object tracking
Andrea Hornakova, Roberto Henschel, Bodo Rosenhahn, and Paul Swoboda · 2020
Later among the works it cites.
Gsm: Graph similarity model for multi-object tracking
Qiankun Liu, Qi Chu, Bin Liu, and Nenghai Yu · 2020
Later among the works it cites.
Tubetk: Adopting tubes to track multi-object in a one-step training model
Bo Pang, Yizhuo Li, Yifan Zhang, Muchen Li, and Cewu Lu · 2020
Later among the works it cites.
Chained-tracker: Chaining paired attentive regression results for end-to-end joint multiple-object detection and tracking
Jinlong Peng, Changan Wang, Fangbin Wan, Yang Wu, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu · 2020
Later among the works it cites.
Learning multi-object tracking and segmentation from automatic annotations
Lorenzo Porzi, Markus Hofinger, Idoia Ruiz, Joan Serrat, Samuel Rota Bulo, and Peter Kontschieder · 2020
Later among the works it cites.
Segment as points for efficient online multi-object tracking and segmentation
Zhenbo Xu, Wei Zhang, Xiao Tan, Wei Yang, Huan Huang, Shilei Wen, Errui Ding, and Liusheng Huang · 2020
Later among the works it cites.
Long-term tracking with deep tracklet association
Y. Zhang, H. Sheng, Y. Wu, S. Wang, W. Lyu, W. Ke, and Z. Xiong · 2020
Later among the works it cites.
Tracking objects as points
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl · 2020
Later among the works it cites.
Quasi-dense similarity learning for multiple object tracking
Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu · 2021
Closest in time.
Improving multiple pedestrian tracking by track management and occlusion handling
Daniel Stadler and Jurgen Beyerer · 2021
Closest in time.
Learning to track with object permanence
Pavel Tokmakov, Jie Li, Wolfram Burgard, and Adrien Gaidon · 2021
Closest in time.
Multiple object tracking with correlation learning
Qiang Wang, Yun Zheng, Pan Pan, and Yinghui Xu · 2021
Closest in time.
A general recurrent tracking framework without real data
Shuai Wang, Hao Sheng, Yang Zhang, Yubin Wu, and Zhang Xiong · 2021
Closest in time.
Joint object detection and multi-object tracking with graph neural networks
Yongxin Wang, Kris Kitani, and Xinshuo Weng · 2021
Closest in time.
End-to-end video instance segmentation with transformers
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia · 2021
Closest in time.
Track to detect and segment: An online multi-object tracker
Jialian Wu, Jiale Cao, Liangchen Song, Yu Wang, Ming Yang, and Junsong Yuan · 2021
Closest in time.
Remot: A model-agnostic refinement for multiple object tracking
Fan Yang, Xin Chang, Sakriani Sakti, Yang Wu, and Satoshi Nakamura · 2021
Closest in time.
Fairmot: On the fairness of detection and re-identification in multiple object tracking
Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu · 2021
Closest in time.
Improving multiple object tracking with single object tracking
Linyu Zheng, Ming Tang, Yingying Chen, Guibo Zhu, Jinqiao Wang, and Hanqing Lu · 2021
Closest in time.
Deformable detr: Deformable transformers for end-to-end object detection
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…