I followed instructions to train with my own custom dataset. However when I try to evaluate using those trained weight that I got from training my dataset, I get this error.

[01/11 13:41:30 yolact.eval]: Loading model... WARNING [01/11 13:41:34 yolact.model.load]: Some parameters required by the model do not exist in the checkpoint, and are initialized as they should be: prediction_layers.0.conf_layer.weight, prediction_layers.0.conf_layer.bias, semantic_seg_conv.weight, semantic_seg_conv.bias [01/11 13:41:34 yolact.eval]: Model loaded. [01/11 13:41:34 yolact.eval]: Converting to TensorRT... [01/11 13:41:34 yolact.eval]: Converting backbone to TensorRT... [01/11 13:41:37 yolact.eval]: Converting protonet to TensorRT... [01/11 13:41:37 yolact.eval]: Converting FPN to TensorRT... [01/11 13:41:37 yolact.eval]: Converting PredictionModule to TensorRT... [01/11 13:41:52 yolact.eval]: Converted to TensorRT. Traceback (most recent call last): File "eval.py", line 1407, in evaluate(net, dataset) File "eval.py", line 877, in evaluate evalimage(net, inp, out) File "eval.py", line 590, in evalimage preds = net(batch, extras=extras)["pred_outs"] File "/home/jetson/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/jetson/yolact_edge/yolact.py", line 1875, in forward outs_wrapper["pred_outs"] = self.detect(pred_outs, extras=extras) File "/home/jetson/yolact_edge/layers/functions/detection.py", line 78, in call result = self.detect(batch_idx, conf_preds, decoded_boxes, mask_data, inst_data, extras) File "/home/jetson/yolact_edge/layers/functions/detection.py", line 105, in detect boxes, masks, classes, scores = self.fast_nms(boxes, masks, scores, self.nms_thresh, self.top_k) File "/home/jetson/yolact_edge/layers/functions/detection.py", line 168, in fast_nms classes = classes[keep] IndexError: too many indices for tensor of dimension 2

1. Do I need a annotations file for calibrations?
2. I am using TensorRT fp16 optimize (--use_fp16_tensorrt) but the network didn't find even one object (without the TensorRT it's works perfect). It is possible that the fp16 don't do the calibration? It look like it from the code (https://github.com/haotian-liu/yolact_edge/blob/662d760f8b2d8b4409d385aaf172e155aaa3a3d8/utils/tensorrt.py#L38)

Thanks

Hi, When I try to train model $ python3 train.py --config=yolact_edge_config --resume=weights/yolact_edge_vid_resnet50_847_50000.pth

I face the below issue :

• File "/home/vahid/env/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler

• _error_if_any_worker_fails()
  

• RuntimeError: DataLoader worker (pid 88428) is killed by signal: Killed.
• The above exception was the direct cause of the following exception:
• Traceback (most recent call last):
• File "train.py", line 707, in

•  train(0, args=args)
  

• File "train.py", line 357, in train

•  datum = next(data_loader_iter)
  

• File "/home/vahid/env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in next

•  data = self._next_data()
  

• File "/home/vahid/env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1068, in _next_data

•  idx, data = self._get_data()
  

• File "/home/vahid/env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1034, in _get_data

•  success, data = self._try_get_data()
  

• File "/home/vahid/env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 885, in _try_get_data

•  raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
  

• RuntimeError: DataLoader worker (pid(s) 88428) exited unexpectedly

Is there any advice ?

I've been trying to add a new backbone to this project for the past several days and I have several questions. Would you mind helping me out?

In my forward(x) I do the following return tuple(outs) where outs has the output of all 31 blocks of my network. In selected_layers I specify [22, 26, -1]. This returns the following error:

Given groups=1, weight of size [256, 512, 1, 1], expected input[8, 1024, 7, 7] to have 512 channels, but got 1024 channels instead

If I replaced selected_layers with just [22, 26], the network trains fine until it hits the following error:

[01/17 19:20:14 yolact.train]: eta: 0:00:00  epoch: 0  iter: 0  B: 8.873  M: 57.962  C: 21.055  S: 59.843  T: 147.732  time: 5.293  data_time: 0.000  lr: 0.000100  max_mem: 4405M
[01/17 19:20:17 yolact.eval]: Computing validation mAP (this may take a while)...

Traceback (most recent call last):
  File "train.py", line 746, in <module>
    train(0, args=args)
  File "train.py", line 642, in train
    compute_validation_map(yolact_net, val_dataset)
  File "train.py", line 734, in compute_validation_map
    eval_script.evaluate(yolact_net, dataset, train_mode=True, train_cfg=cfg)
  File "/content/radar/eval.py", line 1061, in evaluate
    preds = net(batch, extras=extras)["pred_outs"]
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/content/yolact_edge/yolact.py", line 1876, in forward
    outs_wrapper["pred_outs"] = self.detect(pred_outs, extras=extras)
  File "/content/yolact_edge/layers/functions/detection.py", line 74, in __call__
    conf_preds = conf_data.view(batch_size, num_priors, self.num_classes).transpose(2, 1).contiguous()
RuntimeError: shape '[1, 6958, 81]' is invalid for input of size 396900

I'm so close to getting things working but I always seem to hit a new bump in the road. Any help would be greatly appreciated!

First of all thanks to all developers who build the best model.

I have trained Yolact Edge on a single object. When I try to inference using a trained model with around 50 mAP then it starts prediction(prediction very bad) but if the model mAP around 80 or high then it through a bellow error.

Traceback (most recent call last): File "eval.py", line 1246, in evaluate(net, dataset) File "eval.py", line 894, in evaluate evalvideo(net, args.video) File "eval.py", line 777, in evalvideo frame_buffer.put(frame['value'].get()) File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get raise self._value File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, **kwds)) File "eval.py", line 699, in prep_frame return prep_display(preds, frame, None, None, undo_transform=False, class_color=True) File "eval.py", line 167, in prep_display score_threshold = args.score_threshold) File "/home/ubuntu_pc//yolact_edge/layers/output_utils.py", line 103, in postprocess masks = proto_data @ masks.t() RuntimeError: t() expects a tensor with <= 2 dimensions, but self is 3D

I have changed the dimension of tensor 3D to 2D but getting multiple errors.

If anyone has a solution then please share.

Sorry to bother you.. I retrained my custom dataset to detect only person class.

But when I used the model to evaluate,it showed the error about "Expected 3 elements in a list but found 2"...

Did anyone meet the error before ?

I also used --use_fp16_tensorrt command ,and it still get the error.

I think maybe it's overfitting,so it can't transfer tensorrt properly.

Did it possible ?

Here's my config.py

person_dataset=dataset_base.copy({
    'name': 'person',
    
    'train_info': '/home/jason/dataset/mscoco/train2017_person.json',
    'valid_info': '/home/jason/dataset/mscoco/val2017_person.json',
    'class_names':("person",),
    'label_map': {1:1},
})

Thanks for your reading~

Hi,

Thank you for developing this amazing tool.

I recently modified the yolact.py and added one more mask so that the network predicts 2 of them (instead of 1). I am also using a custom dataset. The training works fine but eval.py is showing the following error:

File "eval.py", line 173, in prep_display
    masks = t[3][idx]
IndexError: too many indices for tensor of dimension 3

I checked the size of idx and t[3] to debug the issue and found the following information:

idx: tensor([0, 1, 2, 3, 4])
idx.size(): torch.Size([5])
t[3].size(): torch.Size([100, 480, 640])

It seems that the error is misleading. Anyway, after a while, I realized that in your eval.py, the variable idx is simply :args.top_k. So I changed it but then the output from eval.py was shown incorrectly.

Moreover, the statement masks = t[3][idx] works fine if PredictionModuleTRTWrapper is disabled (just comment 2 lines). I should also mention that the network shows the above error in forward propagation step in eval.py. This leads me to think that the pred_layer is not set properly.

Can you please help me out? Is input_sizes variable is set properly or needs some modification?

Thank you so much.

Dear Haotian Liu, currently I'm trying the yolact_edge trained on custom coco-like dataset for one class ('person'). Contrary to coco the image resolution is 512.

When I run evaluation with TensorRT conversion I get the following error during the protonet conversion:

[02/23 15:26:23 yolact.eval]: Converting protonet to TensorRT...
Traceback (most recent call last):
  File "eval.py", line 1241, in <module>
    convert_to_tensorrt(net, cfg, args, transform=BaseTransform())
  File "/home/oidpsv/yolact_edge/utils/tensorrt.py", line 156, in convert_to_tensorrt
    net.to_tensorrt_protonet(cfg.torch2trt_protonet_int8, calibration_dataset=calibration_protonet_dataset, batch_size=args.trt_batch_size)
  File "/home/oidpsv/yolact_edge/yolact.py", line 1565, in to_tensorrt_protonet
    self.trt_load_if("proto_net", trt_fn, [x], int8_mode, batch_size=batch_size)
  File "/home/oidpsv/yolact_edge/yolact.py", line 1534, in trt_load_if
    module = trt_fn(module, trt_fn_params)
  File "/opt/conda/lib/python3.6/site-packages/torch2trt-0.1.0-py3.6-linux-x86_64.egg/torch2trt/torch2trt.py", line 555, in torch2trto
    engine = builder.build_cuda_engine(network)
  File "/opt/conda/lib/python3.6/site-packages/torch2trt-0.1.0-py3.6-linux-x86_64.egg/torch2trt/calibration.py", line 51, in get_batch
    buffer[i].copy_(tensor)
RuntimeError: The size of tensor a (69) must match the size of tensor b (64) at non-singleton dimension 2

which can be fixed by changing line 1563 in yolact.py from x = torch.ones((1, 256, 69, 69)).cuda() to x = torch.ones((1, 256, 64, 64)).cuda().

The problem is that in this case further evaluation with TensorRT conversion gives zero mAP and processing of images provides empty result (no masks or boxes). Could you please help me? Many thanks.

Hi @haotian-liu,

I'm working on a custom instance segmentation task with three classes. While I get output segmentations on my Jetson Xavier by using tag --disable_tensorrt, there's no output when I run the model on TensorRT.

I'm training a ResNet50 model on my PC and transferring the learned model to Jetson for inference.

Initially, I suspected the error is similar to issue:27 as I got IndexError Warnings when enabling TensorRT. But while debugging I found that except blocks do no harm.

My commented detection.py file:

# This try-except block aims to fix the IndexError that we might encounter when we train on custom datasets and evaluate with TensorRT enabled. See https://github.com/haotian-liu/yolact_edge/issues/27.
       try:
           classes = classes[keep]
           boxes = boxes[keep]
           masks = masks[keep]
           scores = scores[keep]

           print("Passed first Try/Except")

       except IndexError:
           from utils.logging_helper import log_once
           log_once(self, "issue_27_flatten", name="yolact.layers.detect", 
           message="Encountered IndexError as mentioned in https://github.com/haotian-liu/yolact_edge/issues/27. Flattening predictions to avoid error, please verify the outputs. If there are any problems you met related to this, please report an issue.")

           classes = torch.flatten(classes, end_dim=1)
           boxes = torch.flatten(boxes, end_dim=1)
           masks = torch.flatten(masks, end_dim=1)
           scores = torch.flatten(scores, end_dim=1)
           keep = torch.flatten(keep, end_dim=1)

           idx = torch.nonzero(keep, as_tuple=True)[0]
           print(f"\nIdx: {idx}")
           print(f"Idx_min: {idx.min()} and Idx_max: {idx.max()}")

           classes = torch.index_select(classes, 0, idx)
           boxes = torch.index_select(boxes, 0, idx)
           masks = torch.index_select(masks, 0, idx)
           scores = torch.index_select(scores, 0, idx)

       # Only keep the top cfg.max_num_detections highest scores across all classes
       scores, idx = scores.sort(0, descending=True)
       idx = idx[:cfg.max_num_detections]
       scores = scores[:cfg.max_num_detections]

       print(f"\nIdx: {idx}")
       print(f"Idx_min: {idx.min()} and Idx_max: {idx.max()}")

       try:
           print(f"\nInside second Try")

           print(f"Classes: {classes}")
           print(f"Boxes: {boxes}")

           classes = classes[idx]
           print(f"Classes updated: {classes}")

           boxes= boxes[idx]
           print(f"Boxes updated: {boxes}")
           
           masks = masks[idx]

           print(f"Scores: {scores}")
       except IndexError:
           from utils.logging_helper import log_once
           log_once(self, "issue_27_index_select", name="yolact.layers.detect", message="Encountered IndexError as mentioned in https://github.com/haotian-liu/yolact_edge/issues/27. Using `torch.index_select` to avoid error, please verify the outputs. If there are any problems you met related to this, please report an issue.")

           print(f"\nSecond Try/Except")

           classes = torch.index_select(classes, 0, idx)
           boxes = torch.index_select(boxes, 0, idx)
           masks = torch.index_select(masks, 0, idx)

           print(f"Classes updated: {classes}")
           print(f"Boxes updated: {boxes}")
           print(f"Scores: {scores}")

       return boxes, masks, classes, scores

Command that I use to run evaluation:

~/Projects/yolact_edge$ python3 eval.py --config=yolact_edge_config --trained_model=weights/yolact_edge_2115_110000.pth --score_threshold=0.3 --top_k=20 --image=./test_input/020801_2020_11_25_11_54_18.png
[01/27 15:04:38 yolact.eval]: Loading model...
[01/27 15:04:42 yolact.eval]: Model loaded.
[01/27 15:04:42 yolact.eval]: Converting to TensorRT...
[01/27 15:04:42 yolact.eval]: Converting backbone to TensorRT...
[01/27 15:04:44 yolact.eval]: Converting protonet to TensorRT...
[01/27 15:04:44 yolact.eval]: Converting FPN to TensorRT...
[01/27 15:04:44 yolact.eval]: Converting PredictionModule to TensorRT...
[01/27 15:04:55 yolact.eval]: Converted to TensorRT.
WARNING [01/27 15:04:56 yolact.layers.detect]: Encountered IndexError as mentioned in https://github.com/haotian-liu/yolact_edge/issues/27. Flattening predictions to avoid error, please verify the outputs. If there are any problems you met related to this, please report an issue.

Idx: tensor([ 0,  1,  2,  9, 11, 13, 15, 16, 18, 24, 26, 27, 30, 31, 34])
Idx_min: 0 and Idx_max: 34

Idx: tensor([ 0, 10,  1,  2, 11, 12,  5, 13,  6, 14,  7,  3,  8,  9,  4])
Idx_min: 0 and Idx_max: 14

Inside second Try
Classes: tensor([0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2])
Boxes: tensor([[ 0.7628,  0.4625,  0.7711,  0.5278],
        [ 0.9843,  0.7867,  0.9961,  0.8066],
        [ 0.8758,  0.4074,  0.9116,  0.5297],
        [ 0.7709, -0.0089,  0.9846,  0.9502],
        [ 0.7592,  0.1906,  0.7628,  0.2080],
        [ 0.9848,  0.7869,  0.9971,  0.8092],
        [ 0.7709, -0.0089,  0.9846,  0.9502],
        [ 0.7628,  0.4625,  0.7711,  0.5278],
        [ 0.8758,  0.4074,  0.9116,  0.5297],
        [ 0.7592,  0.1906,  0.7628,  0.2080],
        [ 0.7592,  0.1906,  0.7633,  0.2087],
        [ 0.7709, -0.0089,  0.9846,  0.9502],
        [ 0.8753,  0.4015,  0.9084,  0.5413],
        [ 0.9848,  0.7869,  0.9971,  0.8092],
        [ 0.7628,  0.4625,  0.7711,  0.5278]])
WARNING [01/27 15:04:56 yolact.layers.detect]: Encountered IndexError as mentioned in https://github.com/haotian-liu/yolact_edge/issues/27. Using `torch.index_select` to avoid error, please verify the outputs. If there are any problems you met related to this, please report an issue.

Second Try/Except
Classes updated: tensor([0, 2, 0, 0, 2, 2, 1, 2, 1, 2, 1, 0, 1, 1, 0])
Boxes updated: tensor([[ 0.7628,  0.4625,  0.7711,  0.5278],
        [ 0.7592,  0.1906,  0.7633,  0.2087],
        [ 0.9843,  0.7867,  0.9961,  0.8066],
        [ 0.8758,  0.4074,  0.9116,  0.5297],
        [ 0.7709, -0.0089,  0.9846,  0.9502],
        [ 0.8753,  0.4015,  0.9084,  0.5413],
        [ 0.9848,  0.7869,  0.9971,  0.8092],
        [ 0.9848,  0.7869,  0.9971,  0.8092],
        [ 0.7709, -0.0089,  0.9846,  0.9502],
        [ 0.7628,  0.4625,  0.7711,  0.5278],
        [ 0.7628,  0.4625,  0.7711,  0.5278],
        [ 0.7709, -0.0089,  0.9846,  0.9502],
        [ 0.8758,  0.4074,  0.9116,  0.5297],
        [ 0.7592,  0.1906,  0.7628,  0.2080],
        [ 0.7592,  0.1906,  0.7628,  0.2080]])
Scores: tensor([9.8882e-01, 9.0431e-01, 8.8910e-01, 7.1460e-01, 5.0766e-01, 1.0031e-03,
        4.2995e-04, 3.9997e-04, 2.7969e-04, 1.7686e-04, 1.9921e-05, 1.6216e-05,
        5.4856e-06, 1.4189e-06, 1.8612e-07])

Please note: If I add --disable_tensorrt tag, I get results as the code executes in try blocks. I also ensure to remove the cached '.trt' files in ./weights folder. Can you help me here ? Thank you!

hi,when I detect an image,there is a error.The error is :RuntimeError: non-empty 3D or 4D input tensor expected but got ndim: 4, I hope one can help me. Thanks !

Hi

You claimed that yolact edge runs 67fps on RTX 2080 Ti without TensorRT on your paper. But, I tested it on RTX 3070 graphic card and obtained 30fps. I checked that RTX3070 and RTX2080 Ti are quite similar in performance. Also, I am using Windows machine but I believe it is not the reason why the performance is 2x times lower than what you claimed. I appreciate your feedback.

Thanks.

I downloaded the YolactEdge R-101-FPN and some other trained models to execute the evaluation code. But as shown in the image, all the detection results are "rock","showel" and "teeth". could you give me some advice about the problem? Thanks a lot!

Environment:

• OS: Ubuntu 20.04
• GPU: GTX1660 ti
• CUDA Version 11.4.4

Describe the bug Running inference on video with pre-trained weights is stuck in "converting to TensorRT" step

Full logs Log while running comand: !python3 ./yolact_edge/eval.py --trained_model=./yolact_edge/weights/yolact_edge/yolact_edge_54_800000.pth --score_threshold=0.3 --top_k=100 --video={file_path}:{output_path} --calib_images {calib}

After killing the script:

Environment: Colab jupyter file from yolact_edge github (slightly modified to accomodate gdrive imports and some other stuff, but with same working and instalation procedure):

In SSDAugmentation and SSDAugmentationVideo there is an error in the enabling of the rotation augmentation. The line enable_if(cfg.augment_random_flip, RandomRot90()) should be converted in enable_if(cfg.augment_random_rot90, RandomRot90()) in both classes.

https://github.com/haotian-liu/yolact_edge/blob/3f423ede1aeac73dbf86dadfe85af9e288f7f99b/yolact_edge/utils/augmentations.py#L908

https://github.com/haotian-liu/yolact_edge/blob/3f423ede1aeac73dbf86dadfe85af9e288f7f99b/yolact_edge/utils/augmentations.py#L932

NOT AN ISSUE BUT A QUESTION

Is there a documented way on how to run yolact_edge with deepstream?

As far as I know, one would need an .engine file, and .cpp file in order to parse the output of the model and send it to nvinfer and nvosd deepstream plugins.

Any kind of hint will be appreciated. Thanks!