Multiple-Object Tracking with Transformer

What caused the following problem during the test,Thanks! Traceback (most recent call last): File "main_track.py", line 369, in main(args) File "main_track.py", line 292, in main phase='eval', det_val=args.det_val) File "/data1/anaconda3/envs/TransTrack/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context return func(*args, **kwargs) File "/data1/PycharmProjects/TransTrack/engine_track.py", line 156, in evaluate res_track = tracker.step(results[0]) File "/data1/PycharmProjects/TransTrack/models/tracker.py", line 123, in step if track['age'] < self.max_age:

Hi, when I run mot_test.sh on my own video, it gives error

Test:  [ 6120/17958]  eta: 0:36:03  class_error: 100.00  loss: 0.0854 (3.0055)  loss_ce: 0.0187 (0.1138)  loss_bbox: 0.0000 (0.0711)  loss_giou: 0.0000 (0.3172)  loss_ce_0: 0.0126 (0.0945)  loss_bbox_0: 0.0000 (0.0723)  loss_giou_0: 0.0000 (0.3325)  loss_ce_1: 0.0167 (0.1124)  loss_bbox_1: 0.0000 (0.0727)  loss_giou_1: 0.0000 (0.3258)  loss_ce_2: 0.0161 (0.1037)  loss_bbox_2: 0.0000 (0.0725)  loss_giou_2: 0.0000 (0.3240)  loss_ce_3: 0.0132 (0.0974)  loss_bbox_3: 0.0000 (0.0718)  loss_giou_3: 0.0000 (0.3220)  loss_ce_4: 0.0093 (0.1024)  loss_bbox_4: 0.0000 (0.0729)  loss_giou_4: 0.0000 (0.3265)  loss_ce_unscaled: 0.0093 (0.0569)  class_error_unscaled: 100.0000 (27.9203)  loss_bbox_unscaled: 0.0000 (0.0142)  loss_giou_unscaled: 0.0000 (0.1586)  cardinality_error_unscaled: 500.0000 (498.9012)  loss_ce_0_unscaled: 0.0063 (0.0473)  loss_bbox_0_unscaled: 0.0000 (0.0145)  loss_giou_0_unscaled: 0.0000 (0.1662)  cardinality_error_0_unscaled: 500.0000 (498.9012)  loss_ce_1_unscaled: 0.0084 (0.0562)  loss_bbox_1_unscaled: 0.0000 (0.0145)  loss_giou_1_unscaled: 0.0000 (0.1629)  cardinality_error_1_unscaled: 500.0000 (498.9012)  loss_ce_2_unscaled: 0.0081 (0.0518)  loss_bbox_2_unscaled: 0.0000 (0.0145)  loss_giou_2_unscaled: 0.0000 (0.1620)  cardinality_error_2_unscaled: 500.0000 (498.9012)  loss_ce_3_unscaled: 0.0066 (0.0487)  loss_bbox_3_unscaled: 0.0000 (0.0144)  loss_giou_3_unscaled: 0.0000 (0.1610)  cardinality_error_3_unscaled: 500.0000 (498.9012)  loss_ce_4_unscaled: 0.0046 (0.0512)  loss_bbox_4_unscaled: 0.0000 (0.0146)  loss_giou_4_unscaled: 0.0000 (0.1632)  cardinality_error_4_unscaled: 500.0000 (498.9012)  time: 0.1651  data: 0.0022  max mem: 870
Traceback (most recent call last):
  File "main_track.py", line 367, in <module>
    main(args)
  File "main_track.py", line 289, in main
    test_stats, coco_evaluator, res_tracks = evaluate(model, criterion, postprocessors, data_loader_val,
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/projects/TransTrack/engine_track.py", line 152, in evaluate
    res_track = tracker.step(results[0])
  File "/projects/TransTrack/models/tracker.py", line 110, in step
    track['tracking_id'] = tracks[m1]['tracking_id']
KeyError: 'tracking_id'

I went back the 6120th frame and found there was a long sequence with no people for from 5000th frame. This might be somehow related to the error? do you have idea why there is no key of tracking_id here?

Hi, Peize, Thank you for your great work to propose the first transformer-based MOT.

Sadly, My resources are only 4 GPU (TITIAN RTX-24G), so I set 4 batches per machine (keep 16-batch).

python3 -m torch.distributed.launch --nproc_per_node=4 --use_env main_track.py  --output_dir ./output/crowdhuman_mot_trainhalf --dataset_file mot --coco_path mot --batch_size 4  --with_box_refine --resume pre_trained/crowdhuman_final.pth --epochs 20 --lr_drop 10

However, overall performance drop. In particular, MOTA had 1% performance degradation (64.4% vs 65.4%(paper)) Should I adjust other learning parameters? I really want to reproduce the performance with my 4 GPUs. I'd appreciate it if you could give me an advice.

$ python3 main_track.py --output_dir . --dataset_file mot --coco_path mot --batch_size 1 --resume model/619mot17_mot17.pth --eval --with_box_refine --num_queries 500

Test: Total time: 1:00:17 (1.3641 s / it) Accumulating evaluation results... DONE (t=2.52s). IoU metric: bbox Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.541 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.884 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.589 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.080 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.440 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.636

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.050 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.345 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.641 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.223 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.573 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.719 Creating video index for mot.

I'm trying to train the model to reproduce the result. However, I find that the GPU utilization keeps increasing during training process, which may be related to the release of some variables. After using the GPUitl library to monitor which line leads to the issue, I find that output = MSDA.ms_deform_attn_forward( value, value_spatial_shapes, sampling_locations, attention_weights, ctx.im2col_step) in ms_deform_attn_func.py leads to the increasing of the GPU utilization. I'm wondering whether you have meet the same issue?

Hi, I'm trying to train with "sh track_exps/crowdhuman_mot_trainhalf.sh" on MOT20 with the pretrained model "crowdhuman_final.pth" My GPU env is 8 RTX 3090's, but I'm keep getting the below error. Anyone with the same issue? Thanks

Traceback (most recent call last):
  File "main_track.py", line 390, in <module>
    main(args)
  File "main_track.py", line 195, in main
    sampler_val = DistributedVideoSampler(dataset_val, start_id=args.start_id, shuffle=False)
  File "/home/hyeongkyu/projects/TransTrack/datasets/sampler_video_distributed.py", line 41, in __init__
    split_flags = [c[0] for c in chunks]
  File "/home/hyeongkyu/projects/TransTrack/datasets/sampler_video_distributed.py", line 41, in <listcomp>
    split_flags = [c[0] for c in chunks]
IndexError: index 0 is out of bounds for axis 0 with size 0
Traceback (most recent call last):
  File "main_track.py", line 390, in <module>
    main(args)
  File "main_track.py", line 195, in main
    sampler_val = DistributedVideoSampler(dataset_val, start_id=args.start_id, shuffle=False)
  File "/home/hyeongkyu/projects/TransTrack/datasets/sampler_video_distributed.py", line 41, in __init__
    split_flags = [c[0] for c in chunks]
  File "/home/hyeongkyu/projects/TransTrack/datasets/sampler_video_distributed.py", line 41, in <listcomp>
    split_flags = [c[0] for c in chunks]
IndexError: index 0 is out of bounds for axis 0 with size 0
Done (t=2.44s)
creating index...
Traceback (most recent call last):
  File "main_track.py", line 390, in <module>
    main(args)
  File "main_track.py", line 195, in main
    sampler_val = DistributedVideoSampler(dataset_val, start_id=args.start_id, shuffle=False)
  File "/home/hyeongkyu/projects/TransTrack/datasets/sampler_video_distributed.py", line 41, in __init__
    split_flags = [c[0] for c in chunks]
  File "/home/hyeongkyu/projects/TransTrack/datasets/sampler_video_distributed.py", line 41, in <listcomp>
    split_flags = [c[0] for c in chunks]
IndexError: index 0 is out of bounds for axis 0 with size 0

hi, peize, could you provide the training log? I don't have 8 v100 devices for training , so I need to reduce the batch size. your training log surely helps me check the training process.

Best.

In this paper, it is mentioned that the encoder combines the feature map extracted by the backbone from the current frame with the feature map preserved in the previous frame as the input; But in the implementation of the paper, I found that it seems not to do this operation, but to concat the feature of the two same current frame pictures.

Hi~ Thanks for your code and updating! I have a questioin: Now, is it pre-training both detection and tracking by running track_exps/crowdhuman_train.sh ?

Hi Peize,

Thanks for the wonderful work of TransTrack!

I went through your paper and was still a little confused about the learned feature used for object detection. As you said in the paper, a learned feature is a set of parameters. So, what are the parameters? Could you talk about it in more details?

Thanks in advance for your help!

Hello，thanks a lot for sharing the codes !!!

When running the codes, it rasied an error which is ''No module named MultiScaleDeformableAttention''.

The MultiScaleDeformableAttention is used in models/ops/functions/ms_deform_attn_func.py

Is there something that I missed ？ Thanks for replying.

Hello, I'm trying to reproduce results on MOT20, but I got a 5 points drop using the default hyper-parameter from MOT17. Can you provide the training script on MOT20?

Issue #22 https://github.com/PeizeSun/TransTrack/issues/22#issue-884151001

Similar to the bugs faced in the previous closed issue, has anybody figured a way of exporting the model to ONNX? Have you guys tried to create an export script as well and were the problems faced similar to ours? @PeizeSun @ifzhang @simonwu53 @Abrahamon @iFighting

When i test my demo. I find frequently an object is surrounded by more than one regression box. So i find you didn't delete overlap regression box. Why you didn't delete the overlap regression box?

加载671mot17_crowdhuman_mot17.pth模型后，推理的视频中基本没有检测框，当阈值设成0后，才会出现满屏矩形框。 我的参数这样设置的python demo.py --device cuda --video_input videos/palace.mp4 --demo_output output/ --track_thresh 0.4 --resume pretrained/671mot17_crowdhuman_mot17.pth

I trained custom data according to the format of MOT17 label file, using classification labels from 1 to 4, but I got an error.It illustrates "Assertion idx_dim>=0 && idx_dim<index_size && "index out of bounds" failed.", in models/deformable_detrtrack_train.py line 382 "target_classes_onehot.scatter_(2, target_classes.unsqueeze(-1), 1)".Does anyone have the same bug as me?