Multiple Object Tracking as ID Prediction

This is the official PyTorch implementation of our paper:

Multiple Object Tracking as ID Prediction
🎓 Ruopeng Gao, Yijun Zhang, Limin Wang
📧 Primary contact: ruopenggao@gmail.com

🔍 Overview

TL; DR. MOTIP proposes a new perspective to regard the multi-object tracking task as an ID prediction problem. It directly predicts the ID labels for each object in the tracking process, which is more straightforward and effective.

Abstract. In Multiple Object Tracking (MOT), tracking-by-detection methods have stood the test for a long time, which split the process into two parts according to the definition: object detection and association. They leverage robust single-frame detectors and treat object association as a post-processing step through hand-crafted heuristic algorithms and surrogate tasks. However, the nature of heuristic techniques prevents end-to-end exploitation of training data, leading to increasingly cumbersome and challenging manual modification while facing complicated or novel scenarios. In this paper, we regard this object association task as an End-to-End in-context ID prediction problem and propose a streamlined baseline called MOTIP. Specifically, we form the target embeddings into historical trajectory information while considering the corresponding IDs as in-context prompts, then directly predict the ID labels for the objects in the current frame. Thanks to this end-to-end process, MOTIP can learn tracking capabilities straight from training data, freeing itself from burdensome hand-crafted algorithms. Without bells and whistles, our method achieves impressive state-of-the-art performance in complex scenarios like DanceTrack and SportsMOT, and it performs competitively with other transformer-based methods on MOT17. We believe that MOTIP demonstrates remarkable potential and can serve as a starting point for future research.

🔥 News

• 2024.05.06: We release the training code and scripts 🤗. The pre-training scripts will be released later 🔜. Now you can directly download pre-trained weights from the Cloud ☁️.

• 2024.03.28: We release the inference code, you can evaluate the model following the instructions 🎉. Our model weights and logs are available in the Google Drive ☁️.

• 2024.03.26: The paper is released on arXiv, the code will be available in several days. Welcome to watch our repository for the latest updates 📌.

📈 ​Main Results

💃 ​DanceTrack

Method	Training Data	HOTA	DetA	AssA	MOTA	IDF1	URLs
MOTIP	DT	67.5	79.4	57.6	90.3	72.2	model, config, log
MOTIPDAB	DT	70.0	80.8	60.8	91.0	75.1	model, config, log
MOTIP	DT + CH	71.4	81.3	62.8	91.6	76.3	model, config, log
MOTIP	DT* + CH	73.7	82.6	65.9	92.7	78.4	model, config, log

NOTE

1. MOTIP is built upon original Deformable DETR, while MOTIP_DAB is based on DAB-Deformable DETR.
2. DT and CH are the abbreviations of DanceTrack and CrowdHuman respectively.
3. DT^* denotes we utilize both the training and validation set of DanceTrack for training.

🏀 ​SportsMOT

Method	Training Data	HOTA	DetA	AssA	MOTA	IDF1	URLs
MOTIP	Sports	71.9	83.4	62.0	92.9	75.0	model, config, log
MOTIP	Sports*	75.2	86.5	65.4	96.1	78.2	model, config*, log

NOTE

1. Sports is the abbreviation of SportsMOT.
2. Sports^* denotes we utilize both the training and validation set of SportsMOT for training.
3. config^* represents the configuration that can be used for inference. The corresponding training config file has not been uploaded yet.

🚶 ​MOT17

Method	Training Data	HOTA	DetA	AssA	MOTA	IDF1	URLs
MOTIP	MOT17 + CH	59.2	62.0	56.9	75.5	71.2	model, config, log

NOTE

1. CH is the abbreviation of CrowdHuman.

💨 Quick Start

• See INSTALL.md for instructions of installing required components.
• See DATASET.md for datasets download and preparation.
• See PRETRAIN.md for how to get pretrained DETR weights.
• See GET_STARTED.md for how to get started with our MOTIP.
• See MISCELLANEOUS.md for other miscellaneous settings unrelated to the model structure, such as logging.

💐 Acknowledgements

This project is built upon Deformable DETR, MOTR, DAB-Deformable DETR, TrackEval. Thanks to the contributors of these great codebases.

✏️ Citation

If you think this project is helpful, please feel free to leave a ⭐ and cite our paper:

@article{MOTIP,
  title={Multiple Object Tracking as ID Prediction},
  author={Gao, Ruopeng and Zhang, Yijun and Wang, Limin},
  journal={arXiv preprint arXiv:2403.16848},
  year={2024}
}