2020

FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking

Zhang, Yifu, Wang, Chunyu, Wang, Xinggang et al.

Understand

Multi-object tracking (MOT) is an important problem in computer vision which has a wide range of applications.

Formulating MOT as multi-task learning of object detection and re-ID in a single network is appealing since it allows joint optimization of the two tasks and enjoys high computation efficiency.
However, we find that the two tasks tend to compete with each other which need to be carefully addressed.
In particular, previous works usually treat re-ID as a secondary task whose accuracy is heavily affected by the primary detection task.

Built on

Kuhn HW (1955) The hungarian method for the assignment problem. Naval research logistics quarterly 2(1-2):83–97
1955
Earlier work this paper cites.
Kalman RE (1960) A new approach to linear filtering and prediction problems. J Fluids Eng 82(1):35–45
1960
Earlier work this paper cites.
Bernardin K, Stiefelhagen R (2008) Evaluating multiple object tracking performance: the clear mot metrics. EURASIP Journal on Image and Video Processing 2008:1–10
2008
Earlier work this paper cites.
Ess A, Leibe B, Schindler K, Van Gool L (2008) A mobile vision system for robust multi-person tracking. In: CVPR, IEEE, pp 1–8
2008
Earlier work this paper cites.
Felzenszwalb P, McAllester D, Ramanan D (2008) A discriminatively trained, multiscale, deformable part model. In: CVPR, IEEE, pp 1–8
2008
Earlier work this paper cites.
Zhang L, Li Y, Nevatia R (2008) Global data association for multi-object tracking using network flows. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 1–8
2008
Earlier work this paper cites.
Dollár P, Wojek C, Schiele B, Perona P (2009) Pedestrian detection: A benchmark. In: CVPR, IEEE, pp 304–311
2009
Earlier work this paper cites.
Bolme DS, Beveridge JR, Draper BA, Lui YM (2010) Visual object tracking using adaptive correlation filters. In: CVPR, IEEE, pp 2544–2550
2010
Earlier work this paper cites.
Berclaz J, Fleuret F, Turetken E, Fua P (2011) Multiple object tracking using k-shortest paths optimization. IEEE transactions on pattern analysis and machine intelligence 33(9):1806–1819
2011
Earlier work this paper cites.
Zamir AR, Dehghan A, Shah M (2012) Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs. In: European Conference on Computer Vision, Springer, pp 343–356
2012
Earlier work this paper cites.
Milan A, Roth S, Schindler K (2013) Continuous energy minimization for multitarget tracking. IEEE transactions on pattern analysis and machine intelligence 36(1):58–72
2013
Earlier work this paper cites.
Wang C, Wang Y, Yuille AL (2013) An approach to pose-based action recognition. In: CVPR, pp 915–922
2013
Earlier work this paper cites.
Bae SH, Yoon KJ (2014) Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1218–1225
2014
Earlier work this paper cites.
Henriques JF, Caseiro R, Martins P, Batista J (2014) High-speed tracking with kernelized correlation filters. IEEE transactions on pattern analysis and machine intelligence 37(3):583–596
2014
Earlier work this paper cites.
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980
2014
Earlier work this paper cites.
Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: Common objects in context. In: ECCV, Springer, pp 740–755
2014
Earlier work this paper cites.
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556
2014
Earlier work this paper cites.
Wen L, Li W, Yan J, Lei Z, Yi D, Li SZ (2014) Multiple target tracking based on undirected hierarchical relation hypergraph. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1282–1289
2014
Earlier work this paper cites.
Choi W (2015) Near-online multi-target tracking with aggregated local flow descriptor. In: Proceedings of the IEEE international conference on computer vision, pp 3029–3037
2015
Earlier work this paper cites.
Leal-Taixé L, Milan A, Reid I, Roth S, Schindler K (2015) Motchallenge 2015: Towards a benchmark for multi-target tracking. arXiv preprint arXiv:150401942
2015
Earlier work this paper cites.
Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99
2015
Earlier work this paper cites.
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, et al. (2015) Imagenet large scale visual recognition challenge. International journal of computer vision 115(3):211–252
2015
Earlier work this paper cites.
Xiang Y, Alahi A, Savarese S (2015) Learning to track: Online multi-object tracking by decision making. In: ICCV, pp 4705–4713
2015
Earlier work this paper cites.
Bewley A, Ge Z, Ott L, Ramos F, Upcroft B (2016) Simple online and realtime tracking. In: ICIP, IEEE, pp 3464–3468
2016
Earlier work this paper cites.
Han W, Khorrami P, Paine TL, Ramachandran P, Babaeizadeh M, Shi H, Li J, Yan S, Huang TS (2016) Seq-nms for video object detection. arXiv preprint arXiv:160208465
2016
Earlier work this paper cites.
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR, pp 770–778
2016
Earlier work this paper cites.
Kang K, Ouyang W, Li H, Wang X (2016) Object detection from video tubelets with convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 817–825
2016
Earlier work this paper cites.
Milan A, Leal-Taixé L, Reid I, Roth S, Schindler K (2016) Mot16: A benchmark for multi-object tracking. arXiv preprint arXiv:160300831
2016
Earlier work this paper cites.
Ristani E, Solera F, Zou R, Cucchiara R, Tomasi C (2016) Performance measures and a data set for multi-target, multi-camera tracking. In: ECCV, Springer, pp 17–35
2016
Earlier work this paper cites.
Sanchez-Matilla R, Poiesi F, Cavallaro A (2016) Online multi-target tracking with strong and weak detections. In: ECCV, Springer, pp 84–99
2016
Earlier work this paper cites.
Yang F, Choi W, Lin Y (2016) Exploit all the layers: Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2129–2137
2016
Earlier work this paper cites.

Similar

Yu F, Li W, Li Q, Liu Y, Shi X, Yan J (2016) Poi: Multiple object tracking with high performance detection and appearance feature. In: ECCV, Springer, pp 36–42
2016
Cited alongside, same era.
Bae SH, Yoon KJ (2017) Confidence-based data association and discriminative deep appearance learning for robust online multi-object tracking. IEEE transactions on pattern analysis and machine intelligence 40(3):595–610
2017
Cited alongside, same era.
Bochinski E, Eiselein V, Sikora T (2017) High-speed tracking-by-detection without using image information. In: 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), IEEE, pp 1–6
2017
Cited alongside, same era.
Chen L, Ai H, Shang C, Zhuang Z, Bai B (2017) Online multi-object tracking with convolutional neural networks. In: 2017 IEEE International Conference on Image Processing (ICIP), IEEE, pp 645–649
2017
Cited alongside, same era.
Feichtenhofer C, Pinz A, Zisserman A (2017) Detect to track and track to detect. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3038–3046
2017
Cited alongside, same era.
He K, Gkioxari G, Dollár P, Girshick R (2017) Mask r-cnn. In: ICCV, pp 2961–2969
2017
Cited alongside, same era.
Hermans A, Beyer L, Leibe B (2017) In defense of the triplet loss for person re-identification. arXiv preprint arXiv:170307737
2017
Cited alongside, same era.
Kang K, Li H, Xiao T, Ouyang W, Yan J, Liu X, Wang X (2017) Object detection in videos with tubelet proposal networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 727–735
2017
Cited alongside, same era.
Kokkinos I (2017) Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In: CVPR, pp 6129–6138
2017
Cited alongside, same era.
Luo C, Ma C, Wang C, Wang Y (2017) Learning discriminative activated simplices for action recognition. In: AAAI
2017
Cited alongside, same era.
Ranjan R, Patel VM, Chellappa R (2017) Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. T-PAMI 41(1):121–135
2017
Cited alongside, same era.
Sadeghian A, Alahi A, Savarese S (2017) Tracking the untrackable: Learning to track multiple cues with long-term dependencies. In: Proceedings of the IEEE International Conference on Computer Vision, pp 300–311
2017
Cited alongside, same era.

Then

Chu P, Ling H (2019) Famnet: Joint learning of feature, affinity and multi-dimensional assignment for online multiple object tracking. In: ICCV, pp 6172–6181
2019
Later among the works it cites.
Chu P, Fan H, Tan CC, Ling H (2019) Online multi-object tracking with instance-aware tracker and dynamic model refreshment. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, pp 161–170
2019
Later among the works it cites.
Duan K, Bai S, Xie L, Qi H, Huang Q, Tian Q (2019) Centernet: Keypoint triplets for object detection. In: ICCV, pp 6569–6578
2019
Later among the works it cites.
Henschel R, Zou Y, Rosenhahn B (2019) Multiple people tracking using body and joint detections. In: CVPRW, pp 0–0
2019
Later among the works it cites.
Liu S, Johns E, Davison AJ (2019) End-to-end multi-task learning with attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1871–1880
2019
Later among the works it cites.
Mahmoudi N, Ahadi SM, Rahmati M (2019) Multi-target tracking using cnn-based features: Cnnmtt. Multimedia Tools and Applications 78(6):7077–7096
2019
Later among the works it cites.
Sun S, Akhtar N, Song H, Mian AS, Shah M (2019) Deep affinity network for multiple object tracking. IEEE transactions on pattern analysis and machine intelligence
2019
Later among the works it cites.
Tang P, Wang C, Wang X, Liu W, Zeng W, Wang J (2019) Object detection in videos by high quality object linking. IEEE transactions on pattern analysis and machine intelligence 42(5):1272–1278
2019
Later among the works it cites.
Voigtlaender P, Krause M, Osep A, Luiten J, Sekar BBG, Geiger A, Leibe B (2019) Mots: Multi-object tracking and segmentation. In: CVPR, pp 7942–7951
2019
Later among the works it cites.
Xu J, Cao Y, Zhang Z, Hu H (2019) Spatial-temporal relation networks for multi-object tracking. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 3988–3998
2019
Later among the works it cites.
Yang Z, Liu S, Hu H, Wang L, Lin S (2019) Reppoints: Point set representation for object detection. In: ICCV, pp 9657–9666
2019
Later among the works it cites.
Brasó G, Leal-Taixé L (2020) Learning a neural solver for multiple object tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6247–6257
2020
Closest in time.
Cheng B, Xiao B, Wang J, Shi H, Huang TS, Zhang L (2020) Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In: CVPR
2020
Closest in time.
Dendorfer P, Rezatofighi H, Milan A, Shi J, Cremers D, Reid I, Roth S, Schindler K, Leal-Taixé L (2020) Mot20: A benchmark for multi object tracking in crowded scenes. arXiv preprint arXiv:200309003
2020
Closest in time.
Dong Z, Li G, Liao Y, Wang F, Ren P, Qian C (2020) Centripetalnet: Pursuing high-quality keypoint pairs for object detection. In: CVPR, pp 10519–10528
2020
Closest in time.
Fu J, Zong L, Li Y, Li K, Yang B, Liu X (2020) Model adaption object detection system for robot. In: 2020 39th Chinese Control Conference (CCC), IEEE, pp 3659–3664
2020
Closest in time.
Han S, Huang P, Wang H, Yu E, Liu D, Pan X, Zhao J (2020) Mat: Motion-aware multi-object tracking. arXiv preprint arXiv:200904794
2020
Closest in time.
Hornakova A, Henschel R, Rosenhahn B, Swoboda P (2020) Lifted disjoint paths with application in multiple object tracking. In: International Conference on Machine Learning, PMLR, pp 4364–4375
2020
Closest in time.
Liang C, Zhang Z, Lu Y, Zhou X, Li B, Ye X, Zou J (2020) Rethinking the competition between detection and reid in multi-object tracking. arXiv preprint arXiv:201012138
2020
Closest in time.
Lu Z, Rathod V, Votel R, Huang J (2020) Retinatrack: Online single stage joint detection and tracking. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14668–14678
2020
Closest in time.
Pang B, Li Y, Zhang Y, Li M, Lu C (2020) Tubetk: Adopting tubes to track multi-object in a one-step training model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6308–6318
2020
Closest in time.
Peng J, Wang C, Wan F, Wu Y, Wang Y, Tai Y, Wang C, Li J, Huang F, Fu Y (2020) Chained-tracker: Chaining paired attentive regression results for end-to-end joint multiple-object detection and tracking. In: European Conference on Computer Vision, Springer, pp 145–161
2020
Closest in time.
Radosavovic I, Kosaraju RP, Girshick R, He K, Dollár P (2020) Designing network design spaces. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10428–10436
2020
Closest in time.
Shan C, Wei C, Deng B, Huang J, Hua XS, Cheng X, Liang K (2020) Fgagt: Flow-guided adaptive graph tracking. arXiv preprint arXiv:201009015
2020
Closest in time.
Sun P, Jiang Y, Zhang R, Xie E, Cao J, Hu X, Kong T, Yuan Z, Wang C, Luo P (2020) Transtrack: Multiple-object tracking with transformer. arXiv preprint arXiv:201215460
2020
Closest in time.
Wang Z, Zheng L, Liu Y, Li Y, Wang S (2020b) Towards real-time multi-object tracking. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, Springer, pp 107–122
2020
Closest in time.
Zhang Y, Sheng H, Wu Y, Wang S, Lyu W, Ke W, Xiong Z (2020) Long-term tracking with deep tracklet association. IEEE Transactions on Image Processing 29:6694–6706
2020
Closest in time.
Zhou X, Koltun V, Krähenbühl P (2020) Tracking objects as points. In: European Conference on Computer Vision, Springer, pp 474–490
2020
Closest in time.
Pang J, Qiu L, Li X, Chen H, Li Q, Darrell T, Yu F (2021) Quasi-dense similarity learning for multiple object tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 164–173
2021
Closest in time.
Valmadre J, Bewley A, Huang J, Sun C, Sminchisescu C, Schmid C (2021) Local metrics for multi-object tracking. arXiv preprint arXiv:210402631
2021
Closest in time.
Vandenhende S, Georgoulis S, Van Gansbeke W, Proesmans M, Dai D, Van Gool L (2021) Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…