[CVPR2021 Oral] End-to-End Video Instance Segmentation with Transformers

hi！I successfully executed the instructions include in “Install pycocotools” part，but when I perform inference, the above problem occurs. I tried to solve it, but it didn't work. I hope to get your help！

File "inference.py", line 236, in main(args) File "inference.py", line 172, in main model.load_state_dict(state_dict) File "/root/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1044, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for VisTRsegm: Missing key(s) in state_dict: "vistr.backbone.0.body.layer3.6.conv1.weight",

i have done this

git clone https://github.com/youtubevos/cocoapi.git

cd PythonAPI

python setup.py build_ext --inplace

python setup.py build_ext install

still No module named 'pycocotools.ytvos'

pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI' pip install git+https://github.com/youtubevos/cocoapi.git#"egg=pycocotools&subdirectory=PythonAPI"

Looking in indexes: https://repo.huaweicloud.com/repository/pypi/simple, https://pypi.ngc.nvidia.com Collecting git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI Cloning https://github.com/cocodataset/cocoapi.git to /tmp/pip-req-build-x2a4p4jp Running command git clone -q https://github.com/cocodataset/cocoapi.git /tmp/pip-req-build-x2a4p4jp fatal: unable to access 'https://github.com/cocodataset/cocoapi.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated. WARNING: Discarding git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI. Command errored out with exit status 128: git clone -q https://github.com/cocodataset/cocoapi.git /tmp/pip-req-build-x2a4p4jp Check the logs for full command output. ERROR: Command errored out with exit status 128: git clone -q https://github.com/cocodataset/cocoapi.git /tmp/pip-req-build-x2a4p4jp Check the logs for full command output.

Thank you for sharing this great work! I have a question about your classes index. YoutubeVIS has 40 classes and a empty class is involved in VisTR, thus 41 classes in total, right? Then I noticed that in your implementation, the loss_labels() is computed by

target_classes = torch.full(src_logits.shape[:2], self.num_classes, dtype=torch.int64, device=src_logits.device)
target_classes[idx] = target_classes_o
loss_ce = F.cross_entropy(src_logits.transpose(1, 2), target_classes, self.empty_weight)

Since ytvis's class begins from 1 to 40 and I didn't find any re-index operation in your code, I am confused by the empty class here. Do you use 40 represent empty class (you set self.num_classes=40)? Thank you

Thanks for your excellent work. I have one qustion about the inference process. If the length of video is larger than 36, how to link the tracks from diffenrent clips? looking forward to your reply.

For example lr_backbone in paper 1e-4 and in code 1e-5; epoch 10 in paper and in code 18, if we should adjust it as paper or just use the default params to reproduce the results?

Some questions:

1. In Section 3.1, you use d × (T · H · W ) as input size. Why not T × (d · H · W )?
2. CNN feature is reshaped to B×C×THW. However, the input token size is HW × B × C for transformer encoder in code. It seems confusing.

Thank you very much for your great work. I have followed you. Can VisTR be applied to the task of video objects semantic segmentation? What changes do we need to make when migrating your network to the task of video objects semantic segmentation? looking forward to your reply.

Thanks for your open-source code. Ii is very helpful for my research. But when i want to train the model, i find i counldn't find the corresponding file instance_train_sub.json' or 'instance__val_sub.json' in the Codalab page(https://competitions.codalab.org/competitions/20128#participate-get_data). I only download the image data.But, for annotation file, I just got the test.json or val.json or train.json file. And in those json file, I couldn't fine the annotation information.I would appreciate it if you could help me to correct this problem.

Dear authors,

Thanks for your great open-source work. I have a question regarding the training:

In each epoch, the number of iterations equals to the number of images. However, in each iteration, the input is a whole video which contains 36 images. That said, in average, one image is trained 36 times in the same epoch. In general, I think a common way is to set the number of iterations to be equal to the number of videos, such that one image is only seen once in each epoch. I am wondering why there are such many iterations in one epoch? Is it specially set for the method, or just for a convenient implementation?

Thanks a lot! Look forward hearing from you.

Thanks for the work! Besides the pretrained model, can the precomputed results (jsons) also be made available?

I don't have access to a V100, and I would really appreciate having the precomputed results. Thanks again!

Hello! There is some difference between a definition of YouTubeVIS2021 dataset from Codalab (https://competitions.codalab.org/competitions/28988#participate-get_data) and annotation files from its links to download. Where is a block annotation{ "id" : int, "video_id" : int, "category_id" : int, "segmentations" : [RLE or [polygon] or None], "areas" : [float or None], "bboxes" : [[x,y,width,height] or None], "iscrowd" : 0 or 1, } in these json files? How will a model be trained on this data without any information about masks, boxes ant etc? Сan you advise something how to train the model with my own classes and masks?

for j in range(self.num_frames): img_path = os.path.join(str(self.img_folder), self.vid_infosvid]['file_names'][frame_id-inds[j]]) frame_id-inds[j], Why you do this subtraction here and reverse the order of inds before?

when I run the code on the youtube vis 2019 dataset, the issue is as the picture shows, how do I solve this problem? https://drive.google.com/file/d/1k3ikymcD2-3pBlM8SV_236Xgk3KdM3UD/view?usp=sharing

I have run the setup.py in /dcn, but still can't run /dcn/deform_conv.py.

 from . import _C

ImportError: attempted relative import with no known parent package

How can I save this problem?

Hello,

I downloaded youtube vis 2019 dataset. The format is different for your required format.

Therefore, i am wondering why we used your linked website and got different formats.

Is there any tools we should run to get the format you laid out?

Hi @Epiphqny,

I have a question regarding your implementation, specifically the way you pass raw data to the backbone. If the input is a video with 36 frames, how are they being processed by a ResNet-50/101?

In the BackboneBase class, forward method, you are passing the tensor list to a backbone, but the backbone is expecting an input of size [64, 3, 7, 7]. As I understand from the paper you are reshaping the videos to [36x300x540] .. so there should be one more preprocessing step from the videos to the backbone.

Can you shed some light on this extra step?

Thanks!