Hello, Thanks for sharing the source code of nice work!

I have tried the TAO training code (GTR_TAO_DR2101.yaml) but failed full training due to the memory overhead error. It seems the memory usage increases gradually during training, and reaches the max memory limit. As I am currently using A6000 with 48G gpu, it should be enough based on your training spec (4x 32G V100 gpu). Could you give any ideas? My initial solution is to reduce the video length 8 to 2.

Moreover, I cannot find the move_tao_keyframes.py file. Could you please provide this file?

Thanks,

Hello, I'm having trouble running the inference (the "Demo" section in the README). Below is a notebook link showing the setup and error.

Here is the link to the notebook.

Let me know if anything else needs to be provided.

Much appreciated!

Hi @xingyizhou @noahcao Thank you for sharing this work When I'm trying to run the script in my x86 machine in cpu $python demo.py --config-file configs/GTR_TAO_DR2101.yaml --video-input docs/yfcc_v_acef1cb6d38c2beab6e69e266e234f.mp4 --output output/demo_yfcc.mp4 --opts MODEL.WEIGHTS GTR_TAO_DR2101.pth, I'm getting the following error:

Traceback (most recent call last): File "/home/sravan/SAT/Tracker/GTR/demo.py", line 161, in for vis_frame in demo.run_on_video(video): File "/home/sravan/SAT/Tracker/GTR/gtr/predictor.py", line 147, in run_on_video outputs = self.video_predictor(frames) File "/home/sravan/anaconda3/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context return func(*args, **kwargs) File "/home/sravan/SAT/Tracker/GTR/gtr/predictor.py", line 103, in call predictions = self.model(inputs) File "/home/sravan/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/sravan/SAT/Tracker/GTR/gtr/modeling/meta_arch/gtr_rcnn.py", line 61, in forward return self.sliding_inference(batched_inputs) File "/home/sravan/SAT/Tracker/GTR/gtr/modeling/meta_arch/gtr_rcnn.py", line 81, in sliding_inference instances_wo_id = self.inference( File "/home/sravan/SAT/Tracker/GTR/gtr/modeling/meta_arch/custom_rcnn.py", line 107, in inference features = self.backbone(images.tensor) File "/home/sravan/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/sravan/anaconda3/lib/python3.9/site-packages/detectron2/modeling/backbone/fpn.py", line 126, in forward bottom_up_features = self.bottom_up(x) File "/home/sravan/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/sravan/SAT/Tracker/GTR/third_party/CenterNet2/centernet/modeling/backbone/res2net.py", line 630, in forward x = stage(x) File "/home/sravan/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/sravan/anaconda3/lib/python3.9/site-packages/torch/nn/modules/container.py", line 141, in forward input = module(input) File "/home/sravan/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/sravan/SAT/Tracker/GTR/third_party/CenterNet2/centernet/modeling/backbone/res2net.py", line 457, in forward sp = self.convs[i](sp, offset, mask) File "/home/sravan/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/sravan/anaconda3/lib/python3.9/site-packages/detectron2/layers/deform_conv.py", line 474, in forward x = modulated_deform_conv( File "/home/sravan/anaconda3/lib/python3.9/site-packages/detectron2/layers/deform_conv.py", line 211, in forward raise NotImplementedError("Deformable Conv is not supported on CPUs!") NotImplementedError: Deformable Conv is not supported on CPUs!

How can I solve this?

Thanks for releasing this great work. May I ask for more details about the speed evaluation?

For TAO data, as you used the default detectron2, may I know if you count the inference time of detectron2 for 11.2 FPS, or only the GTR inference time? Since the TAO video sampling rate may not be 30 FPS, does it need to consider this factor and transfer the inference speed?

Thanks.

Hi Xingyi,

I believe the guidelines you write at the doc has some issue. To be precise, to directly evaluate on MOT17 by:

python train_net.py --config-file configs/GTR_MOT_FPN.yaml --eval-only MODEL.WEIGHTS  output/GTR_MOT/GTR_MOT_FPN/model_0004999.pth

we will get the error as: gtr.tracking.trackeval.utils.TrackEvalException: GT file not found for sequence: MOT17-02-FRCNN

Besides, to evaluate on the self-splitted half-val, I assumed we need the files "gt_val_half.txt" under the directory of each sequence?

Could you help to double check if your guideline can work fine with the current version and reach the requirement of the TrackEvallib you adopted? I thought you may miss some guidelines about data splitting and preparation?

Hi Xingyi,

Thanks for the wonderful job. I tried to run the training on MOT17 following the guidelines. But I found some potential typos making that doable.

1. Should we rename the MOT17 train to trainval, which is not explained in the prepare datasets doc?
2. Should the datasets for training be ("mot17_halftrain","crowdhuman_train") instead of ("mot17_halftrain","crowdhuman_amodal_train") in the config file? the later one would raises an error of unregistered dataset:

Nice work! Thank you for sharing the code.

Is training of detector and tracker is joint or separate? It seems from the paper (Section 5.2) that the first detector needs to be trained then the detector is frozen and the tracker is finetuned after that? Is that right inference?

Thanks Gurkirt

Hi there! Thanks for your work.

Here I have 2 questions about the version of lvis dataset:

1. Why did you use v1.0 instead of v10.5?
2. Could you please show me the code which re-map the labels of v1 back to v0.5?

Looking forward to your reply!

Hey @xingyizhou ! 👋

Nice work on the global tracking transformer!

I'm from Replicate, where we're trying to make machine learning reproducible. We noticed you have registered an account with us, and this pull request makes it possible to run your model inside a Docker environment, which makes it easier for other people to run it. We're using an open source tool called Cog to make this process easier.

This also means we can make a web page where other people can try out your model! View it here: https://replicate.com/xingyizhou/gtr. The docker file can be found under the tab ‘run model with docker’. The demo makes it easy for anyone to upload a customised video and see the result effortless.

We usually add some examples to the for un-registered users (it looks like the screenshot below), but we'd like to invite you to claim the page so you can own the page, customise the Example gallery as you like, push any future update to the web demo, and we'll feature it on our website and tweet about it too. You can find the 'Claim this model' button on the top of the page.

Thank you!

Thank you for sharing your excellent work. But when I trained a model in a machine with 8 GPUs, I met OSError: [Errno 113] No route to host as follow: I did not enable firewall. Now, I don't know how to solve it.

I tried to run the predictor on the long video (about 100k frames) using:

python demo.py --config-file configs/GTR_TAO_DR2101.yaml --video-input docs/Long video.mp4 --output output/Long video.mp4 --opts MODEL.WEIGHTS models/GTR_TAO_DR2101.pth

But the process always get "Killed". Are there any suggestions on this?

Hello, Thank you for sharing the excellent project!

My lab don't have RTX 6000GPUs (24G, referenced in paper ) but some GTX 3090 GPUs (24G). So I want to train GTR with GTX3090 on the MOT dataset. Howevr, I met out of memory error even with batchsize = 1. Could you give me any idea?

Hi, it seems that you are using Res2Net101 on TAO. I'm wondering that is it necessary to use such a heavy backbone instead of ResNet50? Will the performance decrease rapidly when using a smaller backbone like ResNet50?

Hi, sorry to bother you. But I have a small question about the evaluation on TAO val set.

As we all know, there are 482 categories in LVISv0.5 which are included in TAO. So when using LVISv1.0 as in GTR, is it correct that there are only 296 classes in LVISv1.0 which are included in TAO? I'm not sure whether there are some categories are missed.

The related code is here:

    precisions = lvis_eval.eval['precision']
    assert len(class_names) == precisions.shape[2]
    results_per_category = []
    id2apiid = sorted(lvis_gt.get_cat_ids())
    inst_aware_ap, inst_count = 0, 0
    for idx, name in enumerate(class_names):
        precision = precisions[:, :, idx, 0]
        precision = precision[precision > -1]
        ap = np.mean(precision) if precision.size else float("nan")
        inst_num = len(lvis_gt.get_ann_ids(cat_ids=[id2apiid[idx]]))
        if inst_num > 0:
            results_per_category.append(("{} {}".format(
                name, 
                inst_num if inst_num < 1000 else '{:.1f}k'.format(inst_num / 1000)), 
                float(ap * 100)))
            inst_aware_ap += inst_num * ap
            inst_count += inst_num

During MOT training the input resolution is set to 1280x1280 while the test size is 1560 (longer edge). This mean that the input frames have an aspect-ratio (square) and a resolution (lower) compared to the test ones (rectangular aspect-ratio and bigger resolution). I have tried to test with videos of the same resolution and aspect-ratio of training (1280x1280) but the performances were the worst.

My question is, how is it possible to obtain bad performances while maintaining the aspect-ratio and the same resolution of the training? Shouldn't the network perform better in that situation? If not, what is the reason (maybe I am missing some properties of the detector/transformer module)?

Hi, I cannot find in the detail neither here nor in the paper what are the differences between these two models. The configurations are identical but they differ in the training dataset (half vs full).

In the paper you said: "We follow CenterTrack [68] and split each training sequence in half. We use the first half for training and the second half for validation". But the results in table 3 seems obtained by GTR_MOTFull_FPN.

Which of the two models should be considered the "best" one? May I have more information about this?

Thank you so much in advance.