Dear Author, I am trying to use the model for Refcocog (pre-trained + fine-tuned SeqTR segmentation) and test it on Refcoco dataset and visualize the results.

The code I run is "python tools/inference.py /home/chch3470/SeqTR/configs/seqtr/segmentation/seqtr_segm_refcoco-unc.py "/home/chch3470/SeqTR/work_dir/segm_best.pth" --output-dir="/home/chch3470/SeqTR/attention_map_output" --with-gt --which-set="testA" "

I meet the error below. Do you have any idea why it happens? Is Refcocog (pre-trained + fine-tuned SeqTR segmentation) based on yolo or darknet? If it is based on yolo, what configs should we use? Also, should we change the vis_encs(currently the codebase only provides darknet.py for vis_encs)?

I can visualize the provided models for detection tasks so I guess I know the basic setups...

RuntimeError: Error(s) in loading state_dict for SeqTR: size mismatch for lan_enc.embedding.weight: copying a param with shape torch.Size([12692, 300]) from checkpoint, the shape in current model is torch.Size([10344, 300]). size mismatch for head.transformer.seq_positional_encoding.embedding.weight: copying a param with shape torch.Size([25, 256]) from checkpoint, the shape in current model is torch.Size([37, 256]).

Dear author, Could you please kindly share your versions for each of the following packages? torch, torchvision, mmdet, and mmcv-full

Thank you so much!

Thank author for prviding clear code. When the model trained by use "python tools/train.py configs/seqtr/segmentation/seqtr_mask_[DATASET_NAME].py --cfg-options ema=True", the accuracy function only receives 3 return values, and this cause the training failed. According to post code, the "batch_ie" isn't a significant parameter. It seems a reminder, so I delete the code about "batch_iz" in "./seqtr/api/train.py" that it can work well. Could author gives a description about "batch_ie"? It would be nice if the author provided weights trained for the model. Thank you!

Hi, thanks for the awesome work. Could I ask how could we obtain the token_to_ix.pkl, ix_to_token.pkl, and the word_emb.npz to generate customized dataset? Thank you so much!

作者您好，我正在研读您的code，目前有两个问题存在一些疑问。

1.请问multi-task的训练是detection和segmentation两个任务统一训练吗？还是需要分开训呢？

2.在multi-task的配置文件中，比如 configs/seqtr/multi-task/seqtr_multi-task_refcocog-google.py，其中需要到 '../../base/datasets/multi-task/refcocog-google.py' 的配置文件，但在本项目中没有给出，请问作者会公开这部分的配置吗？或者您可以告诉我该如何更改配置吗？

作者您好，请教您一个问题: seq_in[seq_in != self.end].clamp_(min=0, max=self.end-1) 这句code会将目标bbox的左上角和右下角坐标做一个最大最小值的约束，前提是seq_in != self.num_bin (eg: self.end=1000)，如果碰到刚好seq_in == self.end的情况该怎么办呢？ 即比如seq_in = [806, 59, 1000, 233], self.end=1000, 那么执行上述code时，1000会被过滤掉，不进行约束。同时这是不是就与targets label [X1，Y1，X2，Y2，1000]冲突了，这该怎么解决呢？

麻烦作者有空解答一下，万分感谢！

Hi, thanks for the wonderful work. I am curious about why SeqTR is so memory-efficient. As shown in the config file, SeqTR is trained with a batch size of 128 on a single 32GB GPU! However, for object detectors like DETR, the batch size on each GPU is quite limited. Could you please give some insights about this? Thanks in advance.

Hi, thanks for the awesome work. Datasets and most annotations can be normally downloaded following the README. But I did not find mixed in the provided google drive link. Have I missed something? Thanks in advance.

Hi,

Congratulation!

I want to visualize the attention weights of segmentation points similar to Fig. 5.

According to the paper: "We visualize the cross attention map averaged over decoder layers and attention heads in Fig. 5.", but I am not sure how to incorporate these weights into the original image.

Would you like to share the script or provide a workable idea?

Thanks~

Thanks for your great work! I am new to VG and want to know the source of servel files (ix_to_token.pkl, token_to_ix.pkl and word_emb.npz) below work_dir/data/annotations/dataset_name/. Do you define these vocabs and embeddings yourself or learn from other works? Thanks again!

Hi, here are some questions about multi-task: KeyError: "RefCOCOgUMD: 'GenerateMaskVertices is not in the PIPELINES registry'" Thank you very much for your project and look forward to more code and configuration for multi-task

Hi! The following two functions imshow_expr_bbox, imshow_expr_mask are called in seqtr/apis/inference.py https://github.com/sean-zhuh/SeqTR/blob/36f74bb9da4bcf81775f9f3bb3e54b170860c536/seqtr/apis/inference.py#L6` But I can't find them from seqtr.core. Am I missing anything?

Thanks so much for your help!

Hi, thanks for sharing the great work. I have a question about the is_crowd flag. Why do you need to set it to 1 for multiple masks/ polygons when loading the data? https://github.com/sean-zhuh/SeqTR/blob/36f74bb9da4bcf81775f9f3bb3e54b170860c536/seqtr/datasets/pipelines/loading.py#L126

If it looks like if is_crowd=1, the IoU computation from pycocotool will use a modified criterion that considers the union of gt_mask and pred_mask as pred_mask alone, resulting in a higher number than the standard IoU definition.

https://github.com/sean-zhuh/SeqTR/blob/36f74bb9da4bcf81775f9f3bb3e54b170860c536/seqtr/apis/test.py#L19

(See the note in pycocotool https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/mask.py#L65)

Do I understand this correctly? Thanks for your help!

After completing pre-training, I finetuned to refcoco-unc and found the following error messages File "SeqTR/seqtr/utils/checkpoint.py", line 57, in load_pretrained_checkpoint state, ema_state = ckpt['state_dict'], ckpt['ema_state_dict'] KeyError: 'ema_state_dict' Even after fixing this bug, I still found many bugs (e.g. lan_enc.embedding.weight, model.head) in load_pretrained_checkpoint(). Can you please check it?