I could get the expected results by testing the model on validation set of KITTI dataset. How can I use the MonoRCNN for my own video?

As MonoRCNN is built upon a model from detectron, some of the model parameters don't seem to be recognized by Detectron API when I use demo.py with MonoRCNN Configuration. @Rock-100

I have installed the dependencies and downloaded dateset as explained in README.md While executing the main.py as ./main.py --config-file config/MonoRCNN_KITTI.yaml --num-gpus 1 --resume --eval-only and by setting the VISUALIZE to True. I got output results in /output/evaluation/test/visualization. However, the bounding boxes in output images are completely random and incorrect, program doesn't seem to work correctly.

I don't know how I should approach troubleshooting this issue. My Environment. CUDA 11.0 Python 3.8.5 Torch 1.7.1 I installed Detectrone2 from its official page.

Edit: This part of command line output seems to be the problem.

[05/09 02:15:03 d2.checkpoint.c2_model_loading]: Following weights matched with submodule backbone.bottom_up: | Names in Model | Names in Checkpoint | Shapes | |:------------------|:-------------------------|:------------------------------------------------| | res2.0.conv1.* | res2_0_branch2a_{bn_,w} | (64,) (64,) (64,) (64,) (64,64,1,1) | | res2.0.conv2. | res2_0_branch2b_{bn_,w} | (64,) (64,) (64,) (64,) (64,64,3,3) | | res2.0.conv3. | res2_0_branch2c_{bn_,w} | (256,) (256,) (256,) (256,) (256,64,1,1) | | res2.0.shortcut. | res2_0_branch1_{bn_,w} | (256,) (256,) (256,) (256,) (256,64,1,1) | | res2.1.conv1. | res2_1_branch2a_{bn_,w} | (64,) (64,) (64,) (64,) (64,256,1,1) | | res2.1.conv2. | res2_1_branch2b_{bn_,w} | (64,) (64,) (64,) (64,) (64,64,3,3) | | res2.1.conv3. | res2_1_branch2c_{bn_,w} | (256,) (256,) (256,) (256,) (256,64,1,1) | | res2.2.conv1. | res2_2_branch2a_{bn_,w} | (64,) (64,) (64,) (64,) (64,256,1,1) | | res2.2.conv2. | res2_2_branch2b_{bn_,w} | (64,) (64,) (64,) (64,) (64,64,3,3) | | res2.2.conv3. | res2_2_branch2c_{bn_,w} | (256,) (256,) (256,) (256,) (256,64,1,1) | | res3.0.conv1. | res3_0_branch2a_{bn_,w} | (128,) (128,) (128,) (128,) (128,256,1,1) | | res3.0.conv2. | res3_0_branch2b_{bn_,w} | (128,) (128,) (128,) (128,) (128,128,3,3) | | res3.0.conv3. | res3_0_branch2c_{bn_,w} | (512,) (512,) (512,) (512,) (512,128,1,1) | | res3.0.shortcut. | res3_0_branch1_{bn_,w} | (512,) (512,) (512,) (512,) (512,256,1,1) | | res3.1.conv1. | res3_1_branch2a_{bn_,w} | (128,) (128,) (128,) (128,) (128,512,1,1) | | res3.1.conv2. | res3_1_branch2b_{bn_,w} | (128,) (128,) (128,) (128,) (128,128,3,3) | | res3.1.conv3. | res3_1_branch2c_{bn_,w} | (512,) (512,) (512,) (512,) (512,128,1,1) | | res3.2.conv1. | res3_2_branch2a_{bn_,w} | (128,) (128,) (128,) (128,) (128,512,1,1) | | res3.2.conv2. | res3_2_branch2b_{bn_,w} | (128,) (128,) (128,) (128,) (128,128,3,3) | | res3.2.conv3. | res3_2_branch2c_{bn_,w} | (512,) (512,) (512,) (512,) (512,128,1,1) | | res3.3.conv1. | res3_3_branch2a_{bn_,w} | (128,) (128,) (128,) (128,) (128,512,1,1) | | res3.3.conv2. | res3_3_branch2b_{bn_,w} | (128,) (128,) (128,) (128,) (128,128,3,3) | | res3.3.conv3. | res3_3_branch2c_{bn_,w} | (512,) (512,) (512,) (512,) (512,128,1,1) | | res4.0.conv1. | res4_0_branch2a_{bn_,w} | (256,) (256,) (256,) (256,) (256,512,1,1) | | res4.0.conv2. | res4_0_branch2b_{bn_,w} | (256,) (256,) (256,) (256,) (256,256,3,3) | | res4.0.conv3. | res4_0_branch2c_{bn_,w} | (1024,) (1024,) (1024,) (1024,) (1024,256,1,1) | | res4.0.shortcut. | res4_0_branch1_{bn_,w} | (1024,) (1024,) (1024,) (1024,) (1024,512,1,1) | | res4.1.conv1. | res4_1_branch2a_{bn_,w} | (256,) (256,) (256,) (256,) (256,1024,1,1) | | res4.1.conv2. | res4_1_branch2b_{bn_,w} | (256,) (256,) (256,) (256,) (256,256,3,3) | | res4.1.conv3. | res4_1_branch2c_{bn_,w} | (1024,) (1024,) (1024,) (1024,) (1024,256,1,1) | | res4.2.conv1. | res4_2_branch2a_{bn_,w} | (256,) (256,) (256,) (256,) (256,1024,1,1) | | res4.2.conv2. | res4_2_branch2b_{bn_,w} | (256,) (256,) (256,) (256,) (256,256,3,3) | | res4.2.conv3. | res4_2_branch2c_{bn_,w} | (1024,) (1024,) (1024,) (1024,) (1024,256,1,1) | | res4.3.conv1. | res4_3_branch2a_{bn_,w} | (256,) (256,) (256,) (256,) (256,1024,1,1) | | res4.3.conv2. | res4_3_branch2b_{bn_,w} | (256,) (256,) (256,) (256,) (256,256,3,3) | | res4.3.conv3. | res4_3_branch2c_{bn_,w} | (1024,) (1024,) (1024,) (1024,) (1024,256,1,1) | | res4.4.conv1. | res4_4_branch2a_{bn_,w} | (256,) (256,) (256,) (256,) (256,1024,1,1) | | res4.4.conv2. | res4_4_branch2b_{bn_,w} | (256,) (256,) (256,) (256,) (256,256,3,3) | | res4.4.conv3. | res4_4_branch2c_{bn_,w} | (1024,) (1024,) (1024,) (1024,) (1024,256,1,1) | | res4.5.conv1. | res4_5_branch2a_{bn_,w} | (256,) (256,) (256,) (256,) (256,1024,1,1) | | res4.5.conv2. | res4_5_branch2b_{bn_,w} | (256,) (256,) (256,) (256,) (256,256,3,3) | | res4.5.conv3. | res4_5_branch2c_{bn_,w} | (1024,) (1024,) (1024,) (1024,) (1024,256,1,1) | | res5.0.conv1. | res5_0_branch2a_{bn_,w} | (512,) (512,) (512,) (512,) (512,1024,1,1) | | res5.0.conv2. | res5_0_branch2b_{bn_,w} | (512,) (512,) (512,) (512,) (512,512,3,3) | | res5.0.conv3. | res5_0_branch2c_{bn_,w} | (2048,) (2048,) (2048,) (2048,) (2048,512,1,1) | | res5.0.shortcut. | res5_0_branch1_{bn_,w} | (2048,) (2048,) (2048,) (2048,) (2048,1024,1,1) | | res5.1.conv1. | res5_1_branch2a_{bn_,w} | (512,) (512,) (512,) (512,) (512,2048,1,1) | | res5.1.conv2. | res5_1_branch2b_{bn_,w} | (512,) (512,) (512,) (512,) (512,512,3,3) | | res5.1.conv3. | res5_1_branch2c_{bn_,w} | (2048,) (2048,) (2048,) (2048,) (2048,512,1,1) | | res5.2.conv1. | res5_2_branch2a_{bn_,w} | (512,) (512,) (512,) (512,) (512,2048,1,1) | | res5.2.conv2. | res5_2_branch2b_{bn_,w} | (512,) (512,) (512,) (512,) (512,512,3,3) | | res5.2.conv3. | res5_2_branch2c_{bn_,w} | (2048,) (2048,) (2048,) (2048,) (2048,512,1,1) | | stem.conv1.norm. | res_conv1_bn_* | (64,) (64,) (64,) (64,) | | stem.conv1.weight | conv1_w | (64, 3, 7, 7) |

WARNING [05/09 02:15:04 fvcore.common.checkpoint]: Some model parameters or buffers are not found in the checkpoint:

backbone.bottom_up.res3.0.conv2_offset.{bias, weight} backbone.bottom_up.res3.1.conv2_offset.{bias, weight} backbone.bottom_up.res3.2.conv2_offset.{bias, weight} backbone.bottom_up.res3.3.conv2_offset.{bias, weight} backbone.bottom_up.res4.0.conv2_offset.{bias, weight} backbone.bottom_up.res4.1.conv2_offset.{bias, weight} backbone.bottom_up.res4.2.conv2_offset.{bias, weight} backbone.bottom_up.res4.3.conv2_offset.{bias, weight} backbone.bottom_up.res4.4.conv2_offset.{bias, weight} backbone.bottom_up.res4.5.conv2_offset.{bias, weight} backbone.bottom_up.res5.0.conv2_offset.{bias, weight} backbone.bottom_up.res5.1.conv2_offset.{bias, weight} backbone.bottom_up.res5.2.conv2_offset.{bias, weight} backbone.fpn_lateral2.{bias, weight} backbone.fpn_lateral3.{bias, weight} backbone.fpn_lateral4.{bias, weight} backbone.fpn_lateral5.{bias, weight} backbone.fpn_output2.{bias, weight} backbone.fpn_output3.{bias, weight} backbone.fpn_output4.{bias, weight} backbone.fpn_output5.{bias, weight} proposal_generator.rpn_head.anchor_deltas.{bias, weight} proposal_generator.rpn_head.conv.{bias, weight} proposal_generator.rpn_head.objectness_logits.{bias, weight} roi_heads.att_head.dim_layer.{bias, weight} roi_heads.att_head.fc1.{bias, weight} roi_heads.att_head.fc2.{bias, weight} roi_heads.att_head.kpt_layer.{bias, weight} roi_heads.att_head.yaw_layer.{bias, weight} roi_heads.box_head.fc1.{bias, weight} roi_heads.box_head.fc2.{bias, weight} roi_heads.box_predictor.bbox_pred.{bias, weight} roi_heads.box_predictor.cls_score.{bias, weight} roi_heads.dis_head.H_layer.{bias, weight} roi_heads.dis_head.fc1.{bias, weight} roi_heads.dis_head.fc2.{bias, weight} roi_heads.dis_head.hrec_layer.{bias, weight}

WARNING [05/09 02:15:04 fvcore.common.checkpoint]: The checkpoint state_dict contains keys that are not used by the model:

fc1000.{bias, weight} stem.conv1.bias

[05/09 02:15:04 d2.MonoDet]: Loaded 3769 images in COCO format from ../KITTI/val1/KITTI_val1_val.json

While starting training. The code throws following error. I guess this is because of some mismatch between model parameter and training labels. Reference Has this problem been encountered before? Any insight would be helpful.

[05/10 01:19:15 d2.MonoDet]: Loaded 3712 images in COCO format from ../KITTI/val1/KITTI_val1_train.json
[05/10 01:19:22 d2.MonoDet]: Positive labels: ['Car']
[05/10 01:19:22 d2.MonoDet]: Positive samples: 10081
[05/10 01:19:22 d2.data.build]: Removed 645 images with no usable annotations. 3067 images left.
[05/10 01:19:22 d2.data.common]: Serializing 3067 elements to byte tensors and concatenating them all ...
[05/10 01:19:22 d2.data.common]: Serialized dataset takes 7.84 MiB
[05/10 01:19:22 d2.data.build]: Using training sampler TrainingSampler
[05/10 01:19:22 detectron2]: Starting training from iteration 0
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [109,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [34,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [42,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [49,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [50,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [57,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [4,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [5,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [12,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [19,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [20,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [27,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [64,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [65,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [72,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [79,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [87,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [94,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
  File "./main.py", line 179, in <module>
    launch(
  File "/workspace/detectron2/detectron2/engine/launch.py", line 57, in launch
    main_func(*args)
  File "./main.py", line 174, in main
    do_train(cfg, model, resume=args.resume)
  File "./main.py", line 108, in do_train
    loss_dict = model(data)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/workspace/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 117, in forward
    proposals, proposal_losses = self.proposal_generator(images, features, gt_instances)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/workspace/detectron2/detectron2/modeling/proposal_generator/rpn.py", line 425, in forward
    gt_labels, gt_boxes = self.label_and_sample_anchors(anchors, gt_instances)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/workspace/detectron2/detectron2/modeling/proposal_generator/rpn.py", line 289, in label_and_sample_anchors
    matched_idxs, gt_labels_i = retry_if_cuda_oom(self.anchor_matcher)(match_quality_matrix)
  File "/workspace/detectron2/detectron2/utils/memory.py", line 72, in wrapped
    return func(*args, **kwargs)
  File "/workspace/detectron2/detectron2/modeling/matcher.py", line 88, in __call__
    assert torch.all(match_quality_matrix >= 0)
RuntimeError: CUDA error: device-side assert triggered

Hello, rock-100! Congrats to your work to be accepted by ICCV2021! I am wondering whether you will release your code now that your work has been accepted.