Zsseg.baseline - Zero-Shot Semantic Segmentation

Hello authors, thank you for this great work!

I was trying to run for PASCAL VOC following instructions in the readme. I used the single _single_prompt_ version of config and checked for both CLIP_ADAPTER and REGION_CLIP_ADAPTER these parameters:

PROMPT_LEARNER: predefined
PREDEFINED_PROMPT_TEMPLATES:
    - a sculpture of a {}.

After a single training, for different runs of eval (using the --eval-only and --resume args) with different seeds, I am getting highly varying numbers. For some two runs, I saw a difference of around 20 points for mIOU-unbase. The trained model is not changing so this seemed very strange.

Can you please help me understand if Im making some mistake or if this is expected behaviour, what is the source for this high variation in evaluation? Initial thoughts were that randomly selected prompts were the source, but later checked that it was actually using the single prompt.

Hi,

Thanks for sharing your code.

After training and testing with PASCAL VOC, I am getting very bad masks that do not correspond to the IoU score during the evaluation. For example class "train" IoU is 92 but predicted masks are very poor.

Can you please provide the visualization script?

Thank you.

For evaluation, add --eval-only flag to the traing command. Only --eval-only seems not right. I only get the result of model which is initialized from ImageNet pre-train.

Hi,thanks for sharing your codes. According to the doc, python datasets/prepare_coco_stuff_164k_sem_seg.py datasets/coco I get this error: Traceback (most recent call last): File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "datasets/prepare_coco_stuff_164k_sem_seg.py", line 209, in convert_to_trainID Image.fromarray(mask_copy).save(seg_filename, "PNG") File "/home/qing_chang/env_detetron2/lib/python3.7/site-packages/PIL/Image.py", line 2237, in save fp = builtins.open(filename, "w+b") FileNotFoundError: [Errno 2] No such file or directory: 'datasets/coco/stuffthingmaps_detectron2/val2017/000000263425.png'

Thanks for the great work.

I wonder if the self-training techniques (Sec 6.3 in your paper) are implemented in the codes/commands in https://github.com/MendelXu/zsseg.baseline#training-and-evaluation by default?

This line tries to import RandomSubsetTrainingSampler from detectron2, but this sampler is only available in detectron2 v0.6 (see here), whereas in README instructions the version of detectron2 is 0.5

Is there config file or code file for retrained clip vision encoder? When building the clip model, the parameter of frozen is true. Do I understand correctly?

Thanks for the fantastic paper and codebase! Does the result of training with manually designed prompts and training with learned prompts compare with Table.7 in the paper? In the result of manually desigened prompts, the mIoU of seen and unseen are 28.03 and 57.07 respectively. Useen is better than seen, maybe the dataset has some problem?

Thanks for the great work and the code. I had some doubts regarding the zero-shot training setting. If I understand correctly, in zero-shot setting only the base classes are used for training and the novel classes should be ignored maybe by setting them as ignore labels.

I didn't find any code to handle the novel classes. I was expecting that in the file zsseg.baseline/mask_former/data/dataset_mappers/mask_former_semantic_dataset_mapper.py or zsseg.baseline/mask_former/zero_shot_mask_former_model.py there should be a provision to ignore the novel classes while training.

I can see that you only pass the names of base classes here which are used in text prompts but I don't see any code to modify the groundtruth segmentation masks to ignore the novel classes.

Please let me know if I am missing something or misunderstanding something.

I download the pretrained model Trained Model, and evaluate on the cocostuff dataset. The results of Directly using CLIP vision eccoder without tuning are 21.01 mIoU and 26.95 mIoU for seen and unseen respectively. However, the results of paper are 26.8 mIoU and 29.7 mIoU. Are the results reasonable?

Was there the command of Data Preparetion for Pascal VOC 11k a error?

python datasets/prepare_voc_sem_seg.py datasets/VOC2012
python tools/mask_cls_collect.py datasets/VOC2012/annotations_detectron2/train datasets/VOC2012/annotations_detectron2/train_base_label_count.json

should be

python datasets/prepare_voc_sem_seg.py datasets/VOC2012
python tools/mask_cls_collect.py datasets/VOC2012/annotations_detectron2/train_base datasets/VOC2012/annotations_detectron2/train_base_label_count.json

train -> train_base?

I am a bit confused as to how the VOC2012 dataset should be prepared for this model. The instruction says that the file structure should be like this:

I hope some of my confusions can be cleared up regarding the matter.

1. I'm guessing that the first link is the link to download the whole VOC2012 dataset folder and from there we need to extract the JPEGImages folder and the val.txt file. But the VOC2012/ImageSets folder has 4 separate folders (Action, Layout, Main, Segmentation) under it, each of which contains a val.txt file. Which one am I supposed to select? I was inclined towards selecting the one under the 'Main' folder.
2. The second link, I'm guessing, is a link to download the SegmentationClassAug folder which I am unable to find. Any guidance as to where exactly to look for will be helpful.
3. The third link is one that has a file named trainaug.txt which I am supposed to download and rename as train.txt if I'm not wrong?

Thanks for the fantastic paper and codebase! I ran the manual prompts experiment python train_net.py --config-file configs/coco-stuff-164k-156/zero_shot_maskformer_R101c_single_prompt_bs32_60k.yaml and wanted to know how did you guys visualize your results? It will be great if you can point me in a direction for the same.

Thanks for your great work and released code. I wonder if you can release the pretrained model so that we can evaluate the generalization ability on custom datasets.

Hi there,

Thanks for sharing this good work!

When I am trying to run learned prompt on the Pascal VOC dataset. I use the config file : voc-11k-15/zero_shot_proposal_classification_learn_prompt_bs16_10k.yaml. In this config file, another config file "vo-11k-20/maskformer_R50_bs32_60k.yaml" is required as the base config, however, the base config file is missing. I am wondering if you can update either of these two settings.

Thank you,