I would love to try out this model but I am struggling with installation. The Colab demo does not work either and gives the error:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/content/Mask2Former/mask2former/modeling/pixel_decoder/ops/functions/ms_deform_attn_func.py in <module>()
     21 try:
---> 22     import MultiScaleDeformableAttention as MSDA
     23 except ModuleNotFoundError as e:

ModuleNotFoundError: No module named 'MultiScaleDeformableAttention'

During handling of the above exception, another exception occurred:

ModuleNotFoundError                       Traceback (most recent call last)
7 frames
/content/Mask2Former/mask2former/modeling/pixel_decoder/ops/functions/ms_deform_attn_func.py in <module>()
     27         "\t`sh make.sh`\n"
     28     )
---> 29     raise ModuleNotFoundError(info_string)
     30 
     31 

ModuleNotFoundError: 

Please compile MultiScaleDeformableAttention CUDA op with the following commands:
	`cd mask2former/modeling/pixel_decoder/ops`
	`sh make.sh`


---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

I created a new running environment for mask2former according to the steps. When I train the COCO dataset, I can train normally, but when I train my dataset, I encounter the following problems.

I've been looking for a solution on Google for a long time, so I'd like to ask if you have any similar problems. Thank you very much for your reply.

Hi,

I run Mask2Former on ADE (maskformer2_swin_small_bs16_160k.yaml) with 4 16GB V-100 GPUs. However, I can only achieve 49.6%, which is much worse than the reported result (51.3%). Could you provide the log for me to analysize the result?

Thanks

Hi thank you for your excellent work. I meet a problem when re-run your experiments.

I tried to follow your advice in Getting Started with Mask2Former, and run: python train_net.py --num-gpus 8 \ --config-file configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml after training, the log showed that "Start inference on 625 batches". But after a few days, there are still no new logs. So I kill this process and run python train_net.py \ --config-file configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml \ --eval-only MODEL.WEIGHTS ./output/model_0094999.pth, after evaluation, it showed that the result was was lower that the result from Table 1 in the paper, could you help me see what is the reason for this ^ ^

I‘ve been trying to used it for a nuclei panoptic segmentation task. Dataset is prepared like ADE20K panoptic do. However, in evalutaion, it doesn't proposed any instance after a period time of training.

File "/home/---/anaconda3/envs/mask2former/lib/python3.8/site-packages/panopticapi/evaluation.py", line 224, in pq_compute results[name], per_class_results = pq_stat.pq_average(categories, isthing=isthing) File "/home/---/anaconda3/envs/mask2former/lib/python3.8/site-packages/panopticapi/evaluation.py", line 73, in pq_average return {'pq': pq / n, 'sq': sq / n, 'rq': rq / n, 'n': n}, per_class_results ZeroDivisionError: division by zero

There several possible reasons accounting for it I assume:

• Dataset not well prepared: Are semantic and instance images label folder a must for panoptic? The labeled data I owned is not Detectorn2 format. But I referred to prepare_ade20k_sem_seg, prepare_ade20k_ins_seg and prepare_ade20k_pan_seg. Converted the labeled data to panoptic images (in a folder) and label json file. Commented the line "sem_seg_file_name": sem_label_file, in dataset_dict.
• Configure file not well modified: Another reason maybe model not convergen. Is there any configuration like Mask-RCNN's anchor size or ratio in panopitc segmentation? Because nuclei in whole slide images (crop multiple patches in size 256*256, with one nuclei around (8~16)*(8~16) pixels) is rather small compared to common things in a natural image captioned by camera.

It would be helpful if we could piggy back on your library of pre-trained model for transfer learning. Perhaps this may be accomplished by freezing the first 6 (2L) of 9 layers of Mask2Formers transformer decoder.

Usage may look like this:

export DETECTRON2_DATASETS=/path/to/dir/containing/new/dataset

python train_net.py \
  --config-file <pretrained model config> \
  --pretrained-model /path/to/checkpoint_file
  --transfer-learning-dataset <new dataset>

I've found a really nice backbone ConvNeXt, and I would like to use it with Mask2Former, so that I can achieve a new SOTA result in my own task.

I've changed the code and tested it on the ADE20K dataset with convnext_base_22k_1k_384.pth as backbone weights. I can achieve a significantly better result(mIoU 55.06 VS 52.4(result from your repo)). My training script is as follows:

python train_net.py \
--num-gpus 8 \  
--config-file configs/ade20k/semantic-segmentation/convnext/maskformer2_convnext_base_384_160k_res640.yaml \  
MODEL.WEIGHTS pretraind_model_weights/backbone/convnext_base_22k_1k_384.pkl \  
SOLVER.IMS_PER_BATCH 64 \  
SOLVER.BASE_LR 4e-4 \  
SOLVER.MAX_ITER 50000

Unfortunately, the repo has only released ImageNet-1K/22K pre-trained model. I was wondering If you guys could release the pre-trained model with the coco panoptic segmentation task since the model is too large and we can hardly train from scratch on our personal GPUs.

Thanks!

Hi,

Thanks for your wonderful work and repo.

Could you please provide the instructions on how to visualize the video instance segmentation results on images or videos? Thanks!

Hello! Thank you for sharing. I follow your Example conda environment setup: But when I run the code: It doesnt work successifully. It makes an ImportError

Force cuda since torch ask for a device, not if cuda is in fact available.

torch.cuda.is_available() is misleading since it actually checks for an available device, not if cuda is available. Building on build-nodes where a gpu device is not necessarily available then fails. Force cuda through FORCE_CUDA=1 to allow building.

Hi,

Thank you for this great repository. Can you please advise from where I can download the initial weights mentioned in the config files? For eg: I was not able to find "swin_base_patch4_window12_384.pkl" mentioned in the config file 'Mask2Former/configs/coco/instance-segmentation/swin/maskformer2_swin_base_384_bs16_50ep.yaml' for training from scratch.

I could only find the pre-trained models in Mask2Former Model Zoo.

Thanks.

Hello, I am trying to run mask2former on my local laptop, but it requires a GPU which I do not have. I know that I can run it using Google Colab, but I am unfamiliar with how to use the platform and am not sure if I can manipulate the package in order to visualize the mask binary for the main object in the image. If I do not have a GPU on my computer, what are my options for running mask2former and manipulating the package as I need to? Can you please advise me on what to do in this situation?

Dear authors of the mask2former paper, I just wanted to express my gratitude and appreciation for the excellent work you have done on the mask2former project. Your efforts and contributions have greatly benefited the research community, and I have personally found the insights and results presented in your paper to be incredibly valuable and informative. Thank you for your dedication and commitment to advancing the field. I am really sorry that I have a question, when i run the code with another dataset such as cifar data, i saw the output of segmentation is not good like the coco dataset or maybe because i didn't recognize very well that.

So my question, can anyone provide me a code to visualize the mask binary of the output(panoptic_result, instance_result, semantic_result) to see the main object in order to see the boundaries around the object?

Thank you so much in advance for your answer

visualization of the mask binary of the output(panoptic_result, instance_result, semantic_result) to see the the main object.

Everything works fine with the public COCO dataset. However when using my custom dataset for instance segmentation in a polygon format, when training with Mask2Former some class mAP scores are suspiciously round numbers (e.g. 90.000 mAP) or "NaN".

1. Plotting predictions of the model works fine - every class is plotted correctly and shows the model has learned well
2. Plotting the dataset shows all labels of all classes being correctly plotted by Detectron2 -> no visible issue with the annotations
3. I used Shapely Python package to check the polygons for geometric correctness (polygon.is_valid()) -> every annotation is a valid polygon.

I would be glad to hear how can I pinpoint the issue with the COCOEvaluator

I want to use queries = 50 I also want to use pretrained model(swin-large: queries=200) However, when I python trainnet.py Runtimeerror shows that size mismatch for sem_seg head.predictor.query_feat.weight How to handle this problem, thanks!

when I train mapillary semantic segmentation,error occur below:

Traceback (most recent call last): File "/workspace/Mask2Former/train_net.py", line 322, in launch( File "/workspace/detectron2/detectron2/engine/launch.py", line 82, in launch main_func(*args) File "/workspace/Mask2Former/train_net.py", line 316, in main return trainer.train() File "/workspace/detectron2/detectron2/engine/defaults.py", line 484, in train super().train(self.start_iter, self.max_iter) File "/workspace/detectron2/detectron2/engine/train_loop.py", line 149, in train self.run_step() File "/workspace/detectron2/detectron2/engine/defaults.py", line 494, in run_step self._trainer.run_step() File "/workspace/detectron2/detectron2/engine/train_loop.py", line 413, in run_step loss_dict = self.model(data) File "/root/anaconda3/envs/segment/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/workspace/Mask2Former/mask2former/maskformer_model.py", line 204, in forward targets = self.prepare_targets(gt_instances, images) File "/workspace/Mask2Former/mask2former/maskformer_model.py", line 271, in prepare_targets padded_masks[:, : gt_masks.shape[1], : gt_masks.shape[2]] = gt_masks RuntimeError: expand(CUDABoolType{[21, 1024, 1024, 3]}, size=[21, 1024, 1024]): the number of sizes provided (3) must be greater or equal to the number of dimensions in the tensor (4)

Traceback (most recent call last): File "test.py", line 38, in panoptic_result = v.draw_panoptic_seg(outputs["panoptic_seg"][0].to("cpu"), outputs["panoptic_seg"][1]).get_image() KeyError: 'panoptic_seg'

cfg = get_cfg() add_deeplab_config(cfg) add_maskformer2_config(cfg) #cfg.merge_from_file("configs/coco/panoptic-segmentation/swin/maskformer2_swin_large_IN21k_384_bs16_100ep.yaml") cfg.MODEL.WEIGHTS = './model_final_f07440.pkl' cfg.MODEL.MASK_FORMER.TEST.SEMANTIC_ON = True cfg.MODEL.MASK_FORMER.TEST.INSTANCE_ON = True cfg.MODEL.MASK_FORMER.TEST.PANOPTIC_ON = True predictor = DefaultPredictor(cfg) outputs = predictor(im)

my code

Show panoptic/instance/semantic predictions:

v = Visualizer(im[:, :, ::-1], coco_metadata, scale=1.2, instance_mode=ColorMode.IMAGE_BW) panoptic_result = v.draw_panoptic_seg(outputs["panoptic_seg"][0].to("cpu"), outputs["panoptic_seg"][1]).get_image() v = Visualizer(im[:, :, ::-1], coco_metadata, scale=1.2, instance_mode=ColorMode.IMAGE_BW) instance_result = v.draw_instance_predictions(outputs["instances"].to("cpu")).get_image() v = Visualizer(im[:, :, ::-1], coco_metadata, scale=1.2, instance_mode=ColorMode.IMAGE_BW) semantic_result = v.draw_sem_seg(outputs["sem_seg"].argmax(0).to("cpu")).get_image()