Below is the error message I got. Not so sure about how to fix it. Could you help me with this? Thanks.

==== UPSNet_ROOT$ python upsnet/upsnet_end2end_train.py --cfg upsnet/experiments/upsnet_resnet50_coco.yaml upsnet/../upsnet/config/config.py:180: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. exp_config = edict(yaml.load(f)) Traceback (most recent call last): File "upsnet/upsnet_end2end_train.py", line 61, in from upsnet.models import * File "upsnet/../upsnet/models/init.py", line 1, in from .resnet_upsnet import resnet_50_upsnet, resnet_101_upsnet File "upsnet/../upsnet/models/resnet_upsnet.py", line 22, in from upsnet.models.resnet import get_params, resnet_rcnn, ResNetBackbone File "upsnet/../upsnet/models/resnet.py", line 21, in from upsnet.operators.modules.deform_conv import DeformConv File "upsnet/../upsnet/operators/modules/deform_conv.py", line 22, in from upsnet.operators.functions.deform_conv import DeformConvFunction File "upsnet/../upsnet/operators/functions/deform_conv.py", line 21, in from .._ext.deform_conv import deform_conv_cuda ImportError: upsnet/../upsnet/operators/_ext/deform_conv/deform_conv_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at19UndefinedTensorImpl10_singletonE

I have 50 classes in my dataset out of which 25 are thing and 25 are stuff. in the panoptic_coco_categories.json, i have marked them as isthing = 0 and 1 and i created instance labels for only things and panoptic labels for both things and stuff. during my training, if i write number of classes 25+1 = 26, it does not accept it. but if i write number of classes 51, it trains. what could be the problem?

Did I miss any step to get this error?

~/UPSNet_ROOT$ python upsnet/upsnet_end2end_train.py --cfg upsnet/experiments/upsnet_resnet50_coco.yaml
upsnet/../upsnet/config/config.py:180: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  exp_config = edict(yaml.load(f))
Traceback (most recent call last):
  File "upsnet/upsnet_end2end_train.py", line 60, in <module>
    from upsnet.dataset import *
  File "upsnet/../upsnet/dataset/__init__.py", line 1, in <module>
    from .cityscapes import Cityscapes
  File "upsnet/../upsnet/dataset/cityscapes.py", line 32, in <module>
    from upsnet.dataset.json_dataset import JsonDataset, extend_with_flipped_entries, filter_for_training, add_bbox_regression_targets
  File "upsnet/../upsnet/dataset/json_dataset.py", line 53, in <module>
    import upsnet.bbox.bbox_transform as box_utils
  File "upsnet/../upsnet/bbox/bbox_transform.py", line 15, in <module>
    from .bbox import bbox_overlaps as bbox_overlaps_cython
ModuleNotFoundError: No module named 'upsnet.bbox.bbox'

This is what I got in the route: upsnet/bbox

~/UPSNet_ROOT/upsnet/bbox$ ls
bbox.c                                bbox.pyx            bbox_transform.py  __init__.py  sample_rois.py
bbox.cpython-37m-x86_64-linux-gnu.so  bbox_regression.py  build              __pycache__  setup.py

Thanks.

Hi,

I have ran to this error while I was running (upsnet_resnet50_coco_1gpu.yaml is just a number of gpu change based on 4gpu.yaml

python -u upsnet/upsnet_end2end_train.py --cfg upsnet/experiments/upsnet_resnet50_coco_1gpu.yaml

• running env: pytorch 1.0
• version: 96b7b5172b7b76446f637f4922b7a2054e46703b and this PR change (https://github.com/uber-research/UPSNet/pull/35)

222 2019-05-11 23:08:29,257 | callback.py | line 40 : Batch [1560]  Speed: 2.28 samples/sec Train-rpn_cls_loss=0.202601,    rpn_bbox_loss=0.119188, rcnn_accuracy=0.918098, cls_loss=0.512433,      bbox_loss=0.178919,     mask_loss=0.637334,     fcn_loss=3.486774,      fcn_roi_loss=4.059261,  panoptic_accuracy=0.267040,     panoptic_loss=2.879008,
223 2019-05-11 23:08:38,512 | callback.py | line 40 : Batch [1580]  Speed: 2.16 samples/sec Train-rpn_cls_loss=0.204118,    rpn_bbox_loss=0.121994, rcnn_accuracy=0.918320, cls_loss=0.511436,      bbox_loss=0.178261,     mask_loss=0.637755,     fcn_loss=3.488841,      fcn_roi_loss=4.060041,  panoptic_accuracy=0.265982,     panoptic_loss=2.881260,
224 2019-05-11 23:08:47,145 | callback.py | line 40 : Batch [1600]  Speed: 2.32 samples/sec Train-rpn_cls_loss=0.208637,    rpn_bbox_loss=0.125312, rcnn_accuracy=0.918627, cls_loss=0.510648,      bbox_loss=0.177416,     mask_loss=0.637829,     fcn_loss=3.493248,      fcn_roi_loss=4.063709,  panoptic_accuracy=0.264609,     panoptic_loss=2.885051,
225 2019-05-11 23:08:55,470 | callback.py | line 40 : Batch [1620]  Speed: 2.40 samples/sec Train-rpn_cls_loss=0.210174,    rpn_bbox_loss=0.125260, rcnn_accuracy=0.918797, cls_loss=0.510753,      bbox_loss=0.176945,     mask_loss=0.637923,     fcn_loss=3.496151,      fcn_roi_loss=4.067450,  panoptic_accuracy=0.264234,     panoptic_loss=2.887030,
226 upsnet/../upsnet/operators/modules/fpn_roi_align.py:38: RuntimeWarning: invalid value encountered in sqrt
227   feat_id = np.clip(np.floor(2 + np.log2(np.sqrt(w * h) / 224 + 1e-6)), 0, 3)
228 Traceback (most recent call last):
229   File "upsnet/upsnet_end2end_train.py", line 403, in <module>
230     upsnet_train()
231   File "upsnet/upsnet_end2end_train.py", line 269, in upsnet_train
232     output = train_model(*batch)
233   File "/opt/xxx_workspace/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
234     result = self.forward(*input, **kwargs)
235   File "upsnet/../lib/utils/data_parallel.py", line 110, in forward
236     return self.module(*inputs[0], **kwargs[0])
237   File "/opt/xxx_workspace/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
238     result = self.forward(*input, **kwargs)
239   File "upsnet/../upsnet/models/resnet_upsnet.py", line 139, in forward
240     cls_label, bbox_target, bbox_inside_weight, bbox_outside_weight, mask_target)
241   File "/opt/xxx_workspace/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
242     result = self.forward(*input, **kwargs)
243   File "upsnet/../upsnet/models/rcnn.py", line 190, in forward
244     cls_loss = self.cls_loss(cls_score, cls_label)
245   File "/opt/xxx_workspace/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
246     result = self.forward(*input, **kwargs)
247   File "/opt/xxx_workspace/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 942, in forward
248     ignore_index=self.ignore_index, reduction=self.reduction)
249   File "/opt/xxx_workspace/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2056, in cross_entropy
250     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
251   File "/opt/xxx_workspace/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1869, in nll_loss
252     .format(input.size(0), target.size(0)))
253 ValueError: Expected input batch_size (510) to match target batch_size (512).

2019-04-10 16:16:39,900 | upsnet_end2end_test.py | line 303: unified pano result: Traceback (most recent call last): File "upsnet/upsnet_end2end_test.py", line 312, in upsnet_test() File "upsnet/upsnet_end2end_test.py", line 304, in upsnet_test test_dataset.evaluate_panoptic(test_dataset.get_unified_pan_result(all_ssegs, all_panos, all_pano_cls_inds, stuff_area_limit=config.test.panoptic_stuff_area_limit), os.path.join(final_output_path, 'results', 'pans_unified')) File "upsnet/../upsnet/dataset/base_dataset.py", line 328, in evaluate_panoptic results = pq_compute(gt_json, pred_json, gt_pans, pred_pans, categories) File "upsnet/../upsnet/dataset/base_dataset.py", line 296, in pq_compute results[name], per_class_results = pq_stat.pq_average(categories, isthing=isthing) File "upsnet/../upsnet/dataset/base_dataset.py", line 97, in pq_average return {'pq': pq / n, 'sq': sq / n, 'rq': rq / n, 'n': n}, per_class_results ZeroDivisionError: division by zero

Hi, when I evaluated the trained model, I found this error.

I got this error: OSError: [Errno 12] Cannot allocate memory as below. I am wondering if this is because my current GPU memory is too small, which is 13 GB now.

No matter I run training or testing script, the metrics all I see are AP or AR. Wondering if we should we calculate PQ values ourselves. Or there is anything I missed? Thanks.

When doing an evaluation on the test set, I got the following error. I don't have any clue about it.

2019-04-18 03:56:17,288 | upsnet_end2end_test.py | line 307: unified pano result:
Traceback (most recent call last):
  File "upsnet/upsnet_end2end_test.py", line 316, in <module>
    upsnet_test()
  File "upsnet/upsnet_end2end_test.py", line 308, in upsnet_test
    test_dataset.evaluate_panoptic(test_dataset.get_unified_pan_result(all_ssegs, all_panos, all_pano_cls_inds, stuff_area_limit=config.test.panoptic_stuff_area_limit), os.path.join(final_output_path, 'results', 'pans_unified'))
  File "upsnet/../upsnet/dataset/base_dataset.py", line 333, in evaluate_panoptic
    results = pq_compute(gt_json, pred_json, gt_pans, pred_pans, categories)
  File "upsnet/../upsnet/dataset/base_dataset.py", line 301, in pq_compute
    results[name], per_class_results = pq_stat.pq_average(categories, isthing=isthing)
  File "upsnet/../upsnet/dataset/base_dataset.py", line 97, in pq_average
    return {'pq': pq / n, 'sq': sq / n, 'rq': rq / n, 'n': n}, per_class_results
ZeroDivisionError: division by zero

But, the panoptic segmentation results can be successfully generated like below.

So, how is that possible that I run into the case n = 0? Any idea would be appreciated...Thanks.

Hi,

After following all the steps mentioned and setting up the environment as python 3.6; pytorch 0.4.1; the code has many bugs making it difficult to reproduce the results. Are there some other changes that need to be done - or do following these steps directly work? I am trying to train the model on COCO using the single GPU config file.

Thanks for your great work! I had trained the model with COCO train2017&val2017, and pulled the latest code, but when I trying to test COCO dataset (train2014) with setting the vis_mask=True, I still only got an instance segmentation like figure. Is there anything that needs special attention?

Error message: Traceback (most recent call last): File "upsnet/upsnet_end2end_train.py", line 61, in from upsnet.models import * File "upsnet/../upsnet/models/init.py", line 1, in from .resnet_upsnet import resnet_50_upsnet, resnet_101_upsnet File "upsnet/../upsnet/models/resnet_upsnet.py", line 23, in from upsnet.models.fpn import FPN File "upsnet/../upsnet/models/fpn.py", line 23, in from upsnet.operators.modules.distbatchnorm import BatchNorm2d ModuleNotFoundError: No module named 'upsnet.operators.modules.distbatchnorm'

I have a difficulty in understanding what the dataset structure looks like. Can someone guide me or give a reference where I can read it for myself?

Because the model expects data['im_info'] I don't understand what is im_info contains here. I would really appreciate for any assistance here.

thanks in advance

Hi,

I have been facing this issue from two days, I tried to figure that out but could not come up with a solution. I would really appreciate if someone can guide me where I am going wrong.

The below line calls the RCNN with a list and rois as input.

https://github.com/uber-research/UPSNet/blob/aa8434e5a721ed217849607815304f68dfd7720a/upsnet/models/resnet_upsnet.py#L200

RCNN class in turns calls the roi_pooling and which calls ROIAlignFunction. So the problem is that, the first input from RCNN is the list which is forwarded to all the following function calls. However the ROIAlignFunction expects the first input as variable not as a list (Also extracts its shape in ROIAlignFunction class for further use, which throws an error with list as in input)

Can someone guide me if I have missed out something? Or if this list is converted to a variable somewhere on the way? Thanks in advance.

Hi,

Thank you very much for the great repo.

However, I have hard time in finding the panoptic head code. I can find the code for instance and semantic heads but not for the panoptic. I would like to see the entire architecture of the panoptic head in detail. Can someone guide me in which file exactly I can find the code for panoptic head?

Thanks in advance

Hi,

I would like to get the summary of the model using pytorchsummary. I used below code for the same in upsnet_end2end_train.py

input_shape = [1, 3, 224, 224], summary(train_model, input_shape)

But this throws me an error as below File "../upsnet/models/resnet_upsnet.py", line 90, in forward res2, res3, res4, res5 = self.resnet_backbone(data['data']) TypeError: new(): invalid data type 'str'

https://github.com/uber-research/UPSNet/blob/aa8434e5a721ed217849607815304f68dfd7720a/upsnet/models/resnet_upsnet.py#L90

If someone can guide me what could be the issue? Thanks in advance

Hey Guys,

i'm currently trying to use the networt with coco_4_gpu on infering one image.

I'm inputting: -jpg image with 1280x720 img = cv2.imread('00000001.jpg') img = cv2.resize(img,(1280,720),interpolation=cv2.INTER_CUBIC) img = img.transpose(2, 0, 1) img_tensor = torch.from_numpy(img) img_tensor = torch.unsqueeze(img_tensor,0).type(torch.FloatTensor).cuda() img_info = np.array([[720, 1280, 3]]) data = {'data': img_tensor , 'img_info' : img_info}

When im running the model output i get:

88 fpn_p5_upsample = self.fpn_upsample(fpn_p5_1x1) ---> 89 fpn_p4_plus = fpn_p5_upsample + fpn_p4_1x1 90 fpn_p4_upsample = self.fpn_upsample(fpn_p4_plus) 91 fpn_p3_plus = fpn_p4_upsample + fpn_p3_1x1

RuntimeError: The size of tensor a (46) must match the size of tensor b (45) at non-singleton dimension 2

Whats the matter?

Im on Pytorch 1.4.

Kindest regards!