CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection

Related tags

Deep Learning CLOCs
Overview

CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection

CLOCs is a novel Camera-LiDAR Object Candidates fusion network. It provides a low-complexity multi-modal fusion framework that improves the performance of single-modality detectors. CLOCs operates on the combined output candidates of any 3D and any 2D detector, and is trained to produce more accurate 3D and 2D detection results.

Environment

Tested on python3.6, pytorch 1.1.0, Ubuntu 16.04/18.04.

Performance on KITTI validation set (3712 training, 3769 validation)

CLOCs_SecCas (SECOND+Cascade-RCNN) VS SECOND:

new 40 recall points
Car:      [email protected]       [email protected]   [email protected]
bev:  AP: 96.51 / 95.61, 92.37 / 89.54, 89.41 / 86.96
3d:   AP: 92.74 / 90.97, 82.90 / 79.94, 77.75 / 77.09
old 11 recall points
Car:      [email protected]       [email protected]   [email protected]
bev:  AP: 90.52 / 90.36, 89.29 / 88.10, 87.84 / 86.80
3d:   AP: 89.49 / 88.31, 79.31 / 77.99, 77.36 / 76.52

Install

The code is developed based on SECOND-1.5, please follow the SECOND-1.5 to setup the environment, the dependences for SECOND-1.5 are needed.

pip install shapely fire pybind11 tensorboardX protobuf scikit-image numba pillow

Follow the instructions to install spconv v1.0 (commit 8da6f96). Although CLOCs fusion does not need spconv, but SECOND codebase expects it to be correctly configured.

Then adding the CLOCs directory to your PYTHONPATH, you could add the following line (change '/dir/to/your/CLOCs/' according to your CLOCs directory) in your .bashrc under home directory.

export PYTHONPATH=$PYTHONPATH:'/dir/to/your/CLOCs/'

Prepare dataset (KITTI)

Download KITTI dataset and organize the files as follows:

└── KITTI_DATASET_ROOT
       ├── training    <-- 7481 train data
       |   ├── image_2 <-- for visualization
       |   ├── calib
       |   ├── label_2
       |   ├── velodyne
       |   └── velodyne_reduced <-- empty directory
       └── testing     <-- 7580 test data
       |   ├── image_2 <-- for visualization
       |   ├── calib
       |   ├── velodyne
       |   └── velodyne_reduced <-- empty directory
       └── kitti_dbinfos_train.pkl
       ├── kitti_infos_train.pkl
       ├── kitti_infos_test.pkl
       ├── kitti_infos_val.pkl
       └── kitti_infos_trainval.pkl

Next, you could follow the SECOND-1.5 instructions to create kitti infos, reduced point cloud and groundtruth-database infos, or just download these files from here and put them in the correct directories as shown above.

Fusion of SECOND and Cascade-RCNN

Preparation

CLOCs operates on the combined output of a 3D detector and a 2D detector. For this example, we use SECOND as the 3D detector, Cascade-RCNN as the 2D detector.

  1. For this example, we use detections with sigmoid scores, you could download the Cascade-RCNN detections for the KITTI train and validations set from here file name:'cascade_rcnn_sigmoid_data', or you could run the 2D detector by your self and save the results for the fusion. You could also use your own 2D detector to generate these 2D detections and save them in KITTI format for fusion.

  2. Then download the pretrained SECOND models from here file name: 'second_model.zip', create an empty directory named model_dir under your CLOCs root directory and unzip the files to model_dir. Your CLOCs directory should look like this:

└── CLOCs
       ├── d2_detection_data    <-- 2D detection candidates data
       ├── model_dir       <-- SECOND pretrained weights extracted from 'second_model.zip' 
       ├── second 
       ├── torchplus 
       ├── README.md
  1. Then modify the config file carefully:
train_input_reader: {
  ...
  database_sampler {
    database_info_path: "/dir/to/your/kitti_dbinfos_train.pkl"
    ...
  }
  kitti_info_path: "/dir/to/your/kitti_infos_train.pkl"
  kitti_root_path: "/dir/to/your/KITTI_DATASET_ROOT"
}
...
train_config: {
  ...
  detection_2d_path: "/dir/to/2d_detection/data"
}
...
eval_input_reader: {
  ...
  kitti_info_path: "/dir/to/your/kitti_infos_val.pkl"
  kitti_root_path: "/dir/to/your/KITTI_DATASET_ROOT"
}

Train

python ./pytorch/train.py train --config_path=./configs/car.fhd.config --model_dir=/dir/to/your_model_dir

The trained models and related information will be saved in '/dir/to/your_model_dir'

Evaluation

python ./pytorch/train.py evaluate --config_path=./configs/car.fhd.config --model_dir=/dir/to/your/trained_model --measure_time=True --batch_size=1

For example if you want to test the pretrained model downloaded from here file name: 'CLOCs_SecCas_pretrained.zip', unzip it, then you could run:

python ./pytorch/train.py evaluate --config_path=./configs/car.fhd.config --model_dir=/dir/to/your/CLOCs_SecCas_pretrained --measure_time=True --batch_size=1

If you want to export KITTI format label files, add pickle_result=False at the end of the above commamd.

Fusion of other 3D and 2D detectors

Step 1: Prepare the 2D detection candidates, run your 2D detector and save the results in KITTI format. It is recommended to run inference with NMS score threshold equals to 0 (no score thresholding), but if you don't know how to setup this, it is also fine for CLOCs.

Step 2: Prepare the 3D detection candidates, run your 3D detector and save the results in the format that SECOND could read, including a matrix with shape of N by 7 that contains the N 3D bounding boxes, and a N-element vector for the 3D confidence scores. 7 parameters correspond to the representation of a 3D bounding box. Be careful with the order and coordinate of the 7 parameters, if the parameters are in LiDAR coordinate, the order should be x, y, z, width, length, height, heading; if the parameters are in camera coordinate, the orderr should be x, y, z, lenght, height, width, heading. The details of the transformation functions can be found in file './second/pytorch/core/box_torch_ops.py'.

Step 3: Since the number of detection candidates are different for different 2D/3D detectors, you need to modify the corresponding parameters in the CLOCs code. Then train the CLOCs fusion. For example, there are 70400 (200x176x2) detection candidates in each frame from SECOND with batch size equals to 1. It is a very large number because SECOND is a one-stage detector, for other multi-stage detectors, you could just take the detection candidates before the final NMS function, that would reduce the number of detection candidates to hundreds or thousands.

Step 4: The output of CLOCs are fused confidence scores for all the 3D detection candidates, so you need to replace the old confidence scores (from your 3D detector) with the new fused confidence scores from CLOCs for post processing and evaluation. Then these 3D detection candidates with the corresponding CLOCs fused scores are treated as the input for your 3D detector post processing functions to generate final predictions for evaluation.

Citation

If you find this work useful in your research, please consider citing:

@article{pang2020clocs,
  title={CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection},
  author={Pang, Su and Morris, Daniel and Radha, Hayder},
  booktitle={2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2020}
  organization={IEEE}
}

Acknowledgement

Our code are mainly based on SECOND, thanks for their excellent work!

Comments
  • Can't open file './pytorch/train.py'

    Can't open file './pytorch/train.py'

    Hello,

    I have setup my environment according to the readme, but I cannot run the sample model. Did I do something wrong?

    Any help would be appreciated.

    Thanks.

    opened by harris-mohamed 19
  • Question about fusion of PointRCNN and Cascade R-CNN

    Question about fusion of PointRCNN and Cascade R-CNN

    Hello @pangsu0613! I know you and your lab mate are working for the fusion of other 3D and 2D detectors, but I am really interested in your work and eager to fuse the candidates of PointRCNN and Cascade R-CNN now. I have got the 3D bounding boxes and 3D confidence scores from PointRCNN, but I can not understand your code for I am not familiar with SECOND. Could you please tell me how to modify your code to train the fusion network of PointRCNN and Cascade R-CNN? I am looking forward to your early reply! Thank you in advance!

    opened by rkotimi 13
  • RTX3080 cuda out of memory

    RTX3080 cuda out of memory

    When I run

    python ./pytorch/train.py train --config_path=./configs/car.fhd.config --model_dir=/home/yao/clocs/model_dir/

    Environment Ubuntu20.04 GTX3080 Conda environment:

    Name Version Build Channel

    _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge blas 1.0 mkl
    ca-certificates 2021.5.30 ha878542_0 conda-forge certifi 2021.5.30 py37h89c1867_0 conda-forge cudatoolkit 11.0.221 h6bb024c_0 anaconda cycler 0.10.0 pypi_0 pypi dataclasses 0.6 pypi_0 pypi decorator 4.4.2 pypi_0 pypi easydict 1.9 pypi_0 pypi fire 0.4.0 pypi_0 pypi freetype 2.10.4 h0708190_1 conda-forge future 0.18.2 pypi_0 pypi imageio 2.9.0 pypi_0 pypi intel-openmp 2021.2.0 h06a4308_610
    jpeg 9b h024ee3a_2
    kiwisolver 1.3.1 pypi_0 pypi kornia 0.5.4 pypi_0 pypi lcms2 2.12 h3be6417_0
    ld_impl_linux-64 2.35.1 hea4e1c9_2 conda-forge libffi 3.3 h58526e2_2 conda-forge libgcc-ng 9.3.0 h2828fa1_19 conda-forge libgomp 9.3.0 h2828fa1_19 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libstdcxx-ng 9.3.0 h6de172a_19 conda-forge libtiff 4.2.0 h85742a9_0
    libuv 1.41.0 h7f98852_0 conda-forge libwebp-base 1.2.0 h7f98852_2 conda-forge llvmlite 0.36.0 pypi_0 pypi lz4-c 1.9.3 h9c3ff4c_0 conda-forge matplotlib 3.4.2 pypi_0 pypi mkl 2021.2.0 h06a4308_296
    mkl-service 2.4.0 py37h5e8e339_0 conda-forge mkl_fft 1.3.0 py37h42c9631_2
    mkl_random 1.2.2 py37h219a48f_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge networkx 2.5.1 pypi_0 pypi ninja 1.10.2 h4bd325d_0 conda-forge numba 0.53.1 pypi_0 pypi numpy 1.21.0 pypi_0 pypi numpy-base 1.20.2 py37hfae3a4d_0
    olefile 0.46 pyh9f0ad1d_1 conda-forge opencv-python 4.5.2.54 pypi_0 pypi openssl 1.1.1k h7f98852_0 conda-forge pcdet 0.3.0+65bd4cd dev_0 pillow 8.2.0 py37he98fc37_0
    pip 21.1.2 pyhd8ed1ab_0 conda-forge protobuf 3.17.3 pypi_0 pypi pybind11 2.6.2 pypi_0 pypi pyparsing 2.4.7 pypi_0 pypi python 3.7.10 hffdb5ce_100_cpython conda-forge python-dateutil 2.8.1 pypi_0 pypi python_abi 3.7 1_cp37m conda-forge pywavelets 1.1.1 pypi_0 pypi pyyaml 5.4.1 pypi_0 pypi readline 8.1 h46c0cb4_0 conda-forge scikit-image 0.18.1 pypi_0 pypi scipy 1.7.0 pypi_0 pypi setuptools 49.6.0 py37h89c1867_3 conda-forge shapely 1.7.1 pypi_0 pypi six 1.16.0 pyh6c4a22f_0 conda-forge spconv 1.0 pypi_0 pypi sqlite 3.36.0 h9cd32fc_0 conda-forge tensorboardx 2.2 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi tifffile 2021.6.14 pypi_0 pypi tk 8.6.10 h21135ba_1 conda-forge torch 1.7.0 pypi_0 pypi torchaudio 0.9.0 pypi_0 pypi torchvision 0.8.1+cu101 pypi_0 pypi tqdm 4.61.1 pypi_0 pypi typing_extensions 3.10.0.0 pyha770c72_0 conda-forge wheel 0.36.2 pyhd3deb0d_0 conda-forge xz 5.2.5 h516909a_1 conda-forge zlib 1.2.11 h516909a_1010 conda-forge zstd 1.4.9 ha95c52a_0 conda-forge

    Error: 2d detection path: /home/yao/clocs/d2_detection_data sparse_shape: [ 41 1600 1408] num_class is : 1 load existing model Restoring parameters from ../model_dir/voxelnet-30950.tckpt Restoring parameters from /home/yao/clocs/model_dir/adam_optimizer-1.tckpt {'Car': 5} [-1] load 14357 Car database infos load 2207 Pedestrian database infos load 734 Cyclist database infos load 1297 Van database infos load 56 Person_sitting database infos load 488 Truck database infos load 224 Tram database infos load 337 Misc database infos After filter database: load 10520 Car database infos load 2104 Pedestrian database infos load 594 Cyclist database infos load 826 Van database infos load 53 Person_sitting database infos load 321 Truck database infos load 199 Tram database infos load 259 Misc database infos remain number of infos: 3712 remain number of infos: 3769 WORKER 0 seed: 1624653928 WORKER 1 seed: 1624653929 WORKER 2 seed: 1624653930 /home/yao/clocs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

    getitem(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

    There are 22 candidate implementations:

    • Of which 20 did not match due to: Overload of function 'getitem': File: : Line N/A. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match.
    • Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/typing/arraydecl.py:69

    During: typing of intrinsic-call at /home/yao/clocs/second/core/geometry.py (162)

    File "core/geometry.py", line 162: def points_in_convex_polygon_jit(points, polygon, clockwise=True): vec1 = polygon - polygon[:, [num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)), :] ^

    @numba.jit /home/yao/clocs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

    getitem(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

    There are 22 candidate implementations:

    • Of which 20 did not match due to: Overload of function 'getitem': File: : Line N/A. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match.
    • Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/typing/arraydecl.py:69

    During: typing of intrinsic-call at /home/yao/clocs/second/core/geometry.py (162)

    File "core/geometry.py", line 162: def points_in_convex_polygon_jit(points, polygon, clockwise=True): vec1 = polygon - polygon[:, [num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)), :] ^

    @numba.jit /home/yao/clocs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>

    File "core/geometry.py", line 170: def points_in_convex_polygon_jit(points, polygon, clockwise=True): cross = 0.0 for i in range(num_points): ^

    @numba.jit /home/yao/clocs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

    getitem(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

    There are 22 candidate implementations:

    • Of which 20 did not match due to: Overload of function 'getitem': File: : Line N/A. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match.
    • Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/typing/arraydecl.py:69

    During: typing of intrinsic-call at /home/yao/clocs/second/core/geometry.py (162)

    File "core/geometry.py", line 162: def points_in_convex_polygon_jit(points, polygon, clockwise=True): vec1 = polygon - polygon[:, [num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)), :] ^

    @numba.jit /home/yao/clocs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>

    File "core/geometry.py", line 170: def points_in_convex_polygon_jit(points, polygon, clockwise=True): cross = 0.0 for i in range(num_points): ^

    @numba.jit /home/yao/clocs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>

    File "core/geometry.py", line 170: def points_in_convex_polygon_jit(points, polygon, clockwise=True): cross = 0.0 for i in range(num_points): ^

    @numba.jit /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

    For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

    For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

    For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/typed_passes.py:327: NumbaPerformanceWarning: The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

    To find out why, try turning on parallel diagnostics, see https://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.

    File "utils/eval.py", line 127: @numba.jit(nopython=True,parallel=True) def build_stage2_training(boxes, query_boxes, criterion, scores_3d, scores_2d, dis_to_lidar_3d,overlaps,tensor_index): ^

    state.func_ir.loc)) /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/numba/core/typed_passes.py:327: NumbaPerformanceWarning: The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

    To find out why, try turning on parallel diagnostics, see https://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.

    File "utils/eval.py", line 231: @numba.jit(nopython=True, parallel=True) def d3_box_overlap_kernel(boxes, qboxes, rinc, criterion=-1): ^

    state.func_ir.loc)) /home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.) return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode) Traceback (most recent call last): File "./pytorch/train.py", line 918, in fire.Fire() File "/home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire target=component.name) File "/home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "./pytorch/train.py", line 371, in train raise e File "./pytorch/train.py", line 240, in train all_3d_output_camera_dict, all_3d_output, top_predictions, fusion_input,tensor_index = net(example_torch,detection_2d_path) File "/home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/home/yao/clocs/second/pytorch/models/voxelnet.py", line 310, in forward preds_dict = self.rpn(spatial_features) File "/home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/home/yao/clocs/second/pytorch/models/rpn.py", line 312, in forward x = self.blocksi File "/home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/home/yao/clocs/torchplus/nn/modules/common.py", line 89, in forward input = module(input) File "/home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 178, in forward self.eps, File "/home/yao/Documents/anaconda3/envs/3d/lib/python3.7/site-packages/torch/nn/functional.py", line 2282, in batch_norm input, weight, bias, running_mean, running_var, training, momentum, eps, torch.backends.cudnn.enabled RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 9.78 GiB total capacity; 1.54 GiB already allocated; 36.44 MiB free; 1.57 GiB reserved in total by PyTorch)

    opened by Yaonian72 12
  • Help with inference for own dataset

    Help with inference for own dataset

    Hello, congrats on your nice work. I really enjoyed reading your paper and I am excited to try your code. As I am doing now my master thesis in 3d object detection, I would like to try CLOCs in combination with the state-of-the-art 3D lidar-based detector CenterPoint. Basically, in my project, I have access to camera images and lidar point clouds from sensors that are mounted on a bridge to detect vehicles on the highway, so the sensors are static. I would appreciate it if you could help me with some questions:

    1. Do you think CLOCs is suitable for a real-time application? Only with CenterPoint theoretically, I could achieve 30 FPS

    2. Can you please give me some guideline steps on how to proceed with inference on my custom dataset? Besides first training CenterPoint on my own lidar frames, as stated in the README I should export the detection results in the format that SECOND supports? I guess I should also first train a 2D detector like Cascade R-CNN on my collected image dataset. Afterwards, I could train the CLOCs fusion layer. Once I have this trained model available, how can I use it for inference on unseen data, do you have maybe a sample script for this?

    opened by xavidzo 12
  • Preparation for CLOCs

    Preparation for CLOCs

    Hi, I am planning to do a fusion with Yolov4 and Seconds/PointPillar. Would you be providing a tutorial/guide for the extraction of the bounding boxes before NMS?

    opened by shaunkheng97 12
  • Add a column of data

    Add a column of data

    Hello! @pangsu0613 I read your paper(Fast-CLOCs), and I also added a column of data to your code, changing it from 1p4 to 1p5, but there is an error in my code( self.opt.step() in fastai_optim.py).RuntimeError: The size of tensor a (4) must match the size of tensor b (5) at non-singleton dimension 1.

    opened by xuV2 11
  • Implementation of Cascade RCNN used

    Implementation of Cascade RCNN used

    Hi, Thank you for sharing your work with us.

    For my experiments, I want to modify the Camera (2D Detections) branch and LiDAR (3D Detections) branch before I run them through the CLOCs fusion network.

    1. For the LiDAR branch, I can do it easily by modifying the pre-trained weights provided here before they are loaded into the network.

    2. But for the Camera branch, the current codebase is using the bounding boxes provided here. There are no pre-trained weights available (which I am guessing stems from the fact that the code doesn't use pre-trained weights, instead uses the bounding boxes)

    So, I have the option of producing these bounding boxes with modified weights for which I would like to know which Cascade RCNN implementation(Including the backbone network) was used to produce the current given set of bounding boxes. It would be great if you can let me know about the same.

    opened by abhishektyagi-t 11
  • Considering 3D detection if there is no corresponding 2D detection

    Considering 3D detection if there is no corresponding 2D detection

    Hello @pangsu0613 . In the research paper you have mentioned that you will consider the 3D detection even if there is no corresponding 2D detection. But what happens if you have the 2D detection and there is no corresponding 3D detection.?? image

    I have taken this image from your research paper. Where there is 2D detection but no 3D still its showing the output 3D box. Can you provide clarity on this ??

    One more query:

    1. If we are taking the input as the velodyne infromation to the 3D detections.We need the pickle files like kitti_infos_train for calibration information ??

    2. Car -1 -1 -10 765.27 107.09 816.20 136.49 -1 -1 -1 -1000 -1000 -1000 -10 0.0000 . Can you please also explain this format output from cascaded RCNN which is present in the pickle file for 2D detections??

    3. Is there any straightforward method where we can directly use the 3D detections with out using seconds so that we can directly feed the 2D and 3D detections to clocs and run ?? Thanks....

    opened by vignesh628 9
  • RuntimeError: src/spconv/indice.cu 120 cuda execution failed with error 98

    RuntimeError: src/spconv/indice.cu 120 cuda execution failed with error 98

    ERROR WHILE RUNNING THE INFERENCE

    python /home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py evaluate --config_path=/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/configs/car.fhd.config --model_dir=/home/developer/deep_learning/deepti_ubuntu20/CLOCs/CLOCs_SecCas_pretrained --measure_time=True --batch_size=1

    Predict_test: False sparse_shape: [ 41 1600 1408] num_class is : 1 load existing model load existing model for fusion layer Restoring parameters from /home/developer/deep_learning/deepti_ubuntu20/CLOCs/CLOCs_SecCas_pretrained/fusion_layer-11136.tckpt remain number of infos: 3769 Generate output labels... Traceback (most recent call last): File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py", line 920, in fire.Fire() File "/opt/conda/lib/python3.6/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/opt/conda/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire target=component.name) File "/opt/conda/lib/python3.6/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py", line 658, in evaluate for example in iter(eval_dataloader): File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 517, in next data = self._next_data() File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 557, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/builder/input_reader_builder.py", line 18, in getitem return self._dataset[idx] File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/data/dataset.py", line 70, in getitem prep_func=self._prep_func) File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/data/preprocess.py", line 313, in _read_and_prep_v9 count=-1).reshape([-1, num_point_features]) FileNotFoundError: [Errno 2] No such file or directory: '/home/developer/deep_learning/Projects/KITTI_DATASET_ROOT/KITTI_DATASET_ROOT/training/velodyne_reduced/000001.bin' developer@f0f2e49fc4a2:~/deep_learning/deepti_ubuntu20/CLOCs/second$ developer@f0f2e49fc4a2:~/deep_learning/deepti_ubuntu20/CLOCs/second$ python /home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py evaluate --config_path=/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/configs/car.fhd.config --model_dir=/home/developer/deep_learning/deepti_ubuntu20/CLOCs/CLOCs_SecCas_pretrained --measure_time=True --batch_size=1 Predict_test: False sparse_shape: [ 41 1600 1408] num_class is : 1 load existing model load existing model for fusion layer Restoring parameters from /home/developer/deep_learning/deepti_ubuntu20/CLOCs/CLOCs_SecCas_pretrained/fusion_layer-11136.tckpt remain number of infos: 3769 Generate output labels... /home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

    getitem(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

    There are 22 candidate implementations:

    • Of which 20 did not match due to: Overload of function 'getitem': File: : Line N/A. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match.
    • Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /opt/conda/lib/python3.6/site-packages/numba/core/typing/arraydecl.py:69

    During: typing of intrinsic-call at /home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/core/geometry.py (162)

    File "core/geometry.py", line 162: def points_in_convex_polygon_jit(points, polygon, clockwise=True): vec1 = polygon - polygon[:, [num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)), :] ^

    @numba.jit /home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>

    File "core/geometry.py", line 170: def points_in_convex_polygon_jit(points, polygon, clockwise=True): cross = 0.0 for i in range(num_points): ^

    @numba.jit /opt/conda/lib/python3.6/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /opt/conda/lib/python3.6/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

    For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) Traceback (most recent call last): File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py", line 920, in fire.Fire() File "/opt/conda/lib/python3.6/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/opt/conda/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire target=component.name) File "/opt/conda/lib/python3.6/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py", line 671, in evaluate model_cfg.lidar_input,global_set) File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py", line 462, in predict_kitti_to_anno all_3d_output_camera_dict, all_3d_output, top_predictions, fusion_input,torch_index = net(example,detection_2d_path) File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/models/voxelnet.py", line 304, in forward voxel_features, coors, batch_size_dev) File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/models/middle.py", line 545, in forward ret = self.middle_conv(ret) File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/opt/conda/lib/python3.6/site-packages/spconv/modules.py", line 123, in forward input = module(input) File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/opt/conda/lib/python3.6/site-packages/spconv/conv.py", line 151, in forward self.stride, self.padding, self.dilation, self.output_padding, self.subm, self.transposed, grid=input.grid) File "/opt/conda/lib/python3.6/site-packages/spconv/ops.py", line 89, in get_indice_pairs stride, padding, dilation, out_padding, int(subm), int(transpose)) RuntimeError: /home/developer/deep_learning/deepti_ubuntu20/spconv-8da6f967fb9a054d8870c3515b1b44eca2103634/src/spconv/indice.cu 120 cuda execution failed with error 98

    opened by vignesh628 9
  • error:bboxAP =0,something wrong but I don't no where .I just follow the readme.md introduction. Need help please.

    error:bboxAP =0,something wrong but I don't no where .I just follow the readme.md introduction. Need help please.

    state.func_ir.loc)) Car [email protected], 0.70, 0.70: bbox AP:0.00, 0.00, 0.00 bev AP:0.00, 0.00, 0.00 3d AP:0.00, 0.00, 0.00 Car [email protected], 0.50, 0.50: bbox AP:0.00, 0.00, 0.00 bev AP:0.00, 0.00, 0.00 3d AP:0.00, 0.00, 0.00

    opened by Ri-Bai 8
  • fuse other 2D detection

    fuse other 2D detection

    I have fused other 2D detetions such as yolov5 and yolox, but AP is very low and close to 0. The 3D method is SECOND, and i used the pretrain model mentioned in readme. I only replace the file "d2_detection-data" with yolov5 results in kitti label format. I have no idea about which step is wrong.

    opened by cxy-ccc 6
  • RuntimeError

    RuntimeError

    /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

    getitem(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

    There are 22 candidate implementations:

    Of which 20 did not match due to: Overload of function 'getitem': File: : Line N/A. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match. Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/typing/arraydecl.py:69 During: typing of intrinsic-call at /home/debtor/CLOCs/second/core/geometry.py (162)

    File "core/geometry.py", line 162: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    vec1 = polygon - polygon[:, [num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)), :] ^

    @numba.jit /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

    getitem(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

    There are 22 candidate implementations:

    Of which 20 did not match due to: Overload of function 'getitem': File: : Line N/A. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match. Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/typing/arraydecl.py:69 During: typing of intrinsic-call at /home/debtor/CLOCs/second/core/geometry.py (162)

    File "core/geometry.py", line 162: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    vec1 = polygon - polygon[:, [num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)), :] ^

    @numba.jit /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

    getitem(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

    There are 22 candidate implementations:

    Of which 20 did not match due to: Overload of function 'getitem': File: : Line N/A. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match. Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/typing/arraydecl.py:69 During: typing of intrinsic-call at /home/debtor/CLOCs/second/core/geometry.py (162)

    File "core/geometry.py", line 162: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    vec1 = polygon - polygon[:, [num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)), :] ^

    @numba.jit /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>

    File "core/geometry.py", line 170: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    cross = 0.0 for i in range(num_points): ^

    @numba.jit /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>

    File "core/geometry.py", line 170: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    cross = 0.0 for i in range(num_points): ^

    @numba.jit /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>

    File "core/geometry.py", line 170: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    cross = 0.0 for i in range(num_points): ^

    @numba.jit /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    first convert polygon to directed lines

    num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    first convert polygon to directed lines

    num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    first convert polygon to directed lines

    num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

    For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    first convert polygon to directed lines

    num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

    For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    first convert polygon to directed lines

    num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

    For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    first convert polygon to directed lines

    num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) Traceback (most recent call last): File "./pytorch/train.py", line 918, in fire.Fire() File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire target=component.name) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "./pytorch/train.py", line 371, in train raise e File "./pytorch/train.py", line 240, in train all_3d_output_camera_dict, all_3d_output, top_predictions, fusion_input,tensor_index = net(example_torch,detection_2d_path) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/debtor/CLOCs/second/pytorch/models/voxelnet.py", line 304, in forward voxel_features, coors, batch_size_dev) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/debtor/CLOCs/second/pytorch/models/middle.py", line 545, in forward ret = self.middle_conv(ret) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/spconv/modules.py", line 123, in forward input = module(input) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/spconv/conv.py", line 157, in forward outids.shape[0]) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/spconv/functional.py", line 83, in forward return ops.indice_conv(features, filters, indice_pairs, indice_pair_num, num_activate_out, False, True) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/spconv/ops.py", line 112, in indice_conv int(inverse), int(subm)) RuntimeError: cublas runtime error : the GPU program failed to execute at /tmp/pip-req-build-xea91aoh/aten/src/THC/THCBlas.cu:259

    cuda 11.0,pytorch1.7.1 python3.7

    opened by 15837070577 0
  • when i running train.py

    when i running train.py

    /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

    getitem(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

    There are 22 candidate implementations:

    • Of which 20 did not match due to: Overload of function 'getitem': File: : Line N/A. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match.
    • Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/typing/arraydecl.py:69

    During: typing of intrinsic-call at /home/debtor/CLOCs/second/core/geometry.py (162)

    File "core/geometry.py", line 162: def points_in_convex_polygon_jit(points, polygon, clockwise=True): vec1 = polygon - polygon[:, [num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)), :] ^

    @numba.jit /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

    getitem(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

    There are 22 candidate implementations:

    • Of which 20 did not match due to: Overload of function 'getitem': File: : Line N/A. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match.
    • Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/typing/arraydecl.py:69

    During: typing of intrinsic-call at /home/debtor/CLOCs/second/core/geometry.py (162)

    File "core/geometry.py", line 162: def points_in_convex_polygon_jit(points, polygon, clockwise=True): vec1 = polygon - polygon[:, [num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)), :] ^

    @numba.jit /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

    getitem(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

    There are 22 candidate implementations:

    • Of which 20 did not match due to: Overload of function 'getitem': File: : Line N/A. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match.
    • Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float32, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/typing/arraydecl.py:69

    During: typing of intrinsic-call at /home/debtor/CLOCs/second/core/geometry.py (162)

    File "core/geometry.py", line 162: def points_in_convex_polygon_jit(points, polygon, clockwise=True): vec1 = polygon - polygon[:, [num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)), :] ^

    @numba.jit /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>

    File "core/geometry.py", line 170: def points_in_convex_polygon_jit(points, polygon, clockwise=True): cross = 0.0 for i in range(num_points): ^

    @numba.jit /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>

    File "core/geometry.py", line 170: def points_in_convex_polygon_jit(points, polygon, clockwise=True): cross = 0.0 for i in range(num_points): ^

    @numba.jit /home/debtor/CLOCs/second/core/geometry.py:146: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>

    File "core/geometry.py", line 170: def points_in_convex_polygon_jit(points, polygon, clockwise=True): cross = 0.0 for i in range(num_points): ^

    @numba.jit /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

    For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

    For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) /home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

    For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

    File "core/geometry.py", line 157: def points_in_convex_polygon_jit(points, polygon, clockwise=True): # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^

    state.func_ir.loc)) Traceback (most recent call last): File "./pytorch/train.py", line 918, in fire.Fire() File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire target=component.name) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "./pytorch/train.py", line 371, in train raise e File "./pytorch/train.py", line 240, in train all_3d_output_camera_dict, all_3d_output, top_predictions, fusion_input,tensor_index = net(example_torch,detection_2d_path) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/debtor/CLOCs/second/pytorch/models/voxelnet.py", line 304, in forward voxel_features, coors, batch_size_dev) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/debtor/CLOCs/second/pytorch/models/middle.py", line 545, in forward ret = self.middle_conv(ret) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/spconv/modules.py", line 123, in forward input = module(input) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/spconv/conv.py", line 157, in forward outids.shape[0]) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/spconv/functional.py", line 83, in forward return ops.indice_conv(features, filters, indice_pairs, indice_pair_num, num_activate_out, False, True) File "/home/debtor/anaconda3/envs/CLOCs/lib/python3.6/site-packages/spconv/ops.py", line 112, in indice_conv int(inverse), int(subm)) RuntimeError: cublas runtime error : the GPU program failed to execute at /tmp/pip-req-build-xea91aoh/aten/src/THC/THCBlas.cu:259

    opened by Debtor141 0
  • Adding temporal data to clocs_SecCas

    Adding temporal data to clocs_SecCas

    Hi @pangsu0613 I was trying some experiment to pass image and pointcloud data from previous time frames to clocs_SecCas. And modifying by adding some recurrent layers in clocs. I want to check whether this leads to improved detections accuracy. Will this be possible in the current architecture and do you have any suggestion about which dataset I can use for this kind of experiment.

    opened by Anirbanbhk88 19
  • Regrading Pedestrian and Cyclist detection

    Regrading Pedestrian and Cyclist detection

    Hi @pangsu0613 Thanks for sharing your work. I am performing my task in sensor fusion based on the CLOCs paper. So I am trying to implement it. I am able to get the same results that you have posted for while Cars. I am now trying to perform the detections for cyclists and pedestrians. In the cyclist/pedestrian detection section you have mentioned:

     **Step1:Prepare the config files** 
    Under the ./configs directory, edit the file pedestrian.fhd.config or cyclist.fhd.config:
    
    - Change the value of detection_2d_path to your pedestrian 2D detections.
    - Set steps and steps_per_eval to your desired value (tweak based on training set size, for example, if we have 3712 training examples, setting “steps” equal to 74240 (3712 x 20) means training for 20 epochs. “steps_per_eval” means the step interval for each evaluation, for example, if you set “steps_per_eval” to 3712, it means doing evaluation for each epoch.)
    
    

    For the step Change the value of detection_2d_path to your pedestrian 2D detections. I could not find any pedestrian 2D detections in the drive link Do we need to train the Cascade-RCNN separately and generate the 2D detections for pedestrian and cyclist. Or else could you share them?

    opened by Anirbanbhk88 5
Owner
Su Pang
PhD working in autonomous vehicles
Su Pang
This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection, built on SECOND.

3D-CVF This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object

YecheolKim 97 Dec 20, 2022
Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal, multi-exposure and multi-focus image fusion.

U2Fusion Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal (VIS-IR, medical), multi

Han Xu 129 Dec 11, 2022
An efficient and effective learning to rank algorithm by mining information across ranking candidates. This repository contains the tensorflow implementation of SERank model. The code is developed based on TF-Ranking.

SERank An efficient and effective learning to rank algorithm by mining information across ranking candidates. This repository contains the tensorflow

Zhihu 44 Oct 20, 2022
Code for "FPS-Net: A convolutional fusion network for large-scale LiDAR point cloud segmentation".

FPS-Net Code for "FPS-Net: A convolutional fusion network for large-scale LiDAR point cloud segmentation", accepted by ISPRS journal of Photogrammetry

null 15 Nov 30, 2022
Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments

Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments Paper: arXiv (ICRA 2021) Video : https://youtu.be/CC

Sachini Herath 68 Jan 3, 2023
Automatic Calibration for Non-repetitive Scanning Solid-State LiDAR and Camera Systems

ACSC Automatic extrinsic calibration for non-repetitive scanning solid-state LiDAR and camera systems. System Architecture 1. Dependency Tested with U

KINO 192 Dec 13, 2022
Camera-caps - Examine the camera capabilities for V4l2 cameras

camera-caps This is a graphical user interface over the v4l2-ctl command line to

Jetsonhacks 25 Dec 26, 2022
Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera.

Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera. This project prepares training and testing data for various deep learning projects such as 6D object pose estimation projects singleshotpose, as well as object detection and instance segmentation projects.

null 305 Dec 16, 2022
Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather

LiDAR fog simulation Created by Martin Hahner at the Computer Vision Lab of ETH Zurich. This is the official code release of the paper Fog Simulation

Martin Hahner 110 Dec 30, 2022
OpenPCDet Toolbox for LiDAR-based 3D Object Detection.

OpenPCDet OpenPCDet is a clear, simple, self-contained open source project for LiDAR-based 3D object detection. It is also the official code release o

OpenMMLab 3.2k Dec 31, 2022
LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection

LiDAR Distillation Paper | Model LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection Yi Wei, Zibu Wei, Yongming Rao, Jiax

Yi Wei 75 Dec 22, 2022
LiDAR R-CNN: An Efficient and Universal 3D Object Detector

LiDAR R-CNN: An Efficient and Universal 3D Object Detector Introduction This is the official code of LiDAR R-CNN: An Efficient and Universal 3D Object

TuSimple 295 Jan 5, 2023
Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Data

LiDAR-MOS: Moving Object Segmentation in 3D LiDAR Data This repo contains the code for our paper: Moving Object Segmentation in 3D LiDAR Data: A Learn

Photogrammetry & Robotics Bonn 394 Dec 29, 2022
A general python framework for single object tracking in LiDAR point clouds, based on PyTorch Lightning.

Open3DSOT A general python framework for single object tracking in LiDAR point clouds, based on PyTorch Lightning. The official code release of BAT an

Kangel Zenn 172 Dec 23, 2022
Official implement of Paper:A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sening images

A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sensing images 深度监督影像融合网络DSIFN用于高分辨率双时相遥感影像变化检测 Of

Chenxiao Zhang 135 Dec 19, 2022
[TIP 2020] Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion

Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion Code for Multi-Temporal Scene Classification and Scene Ch

Lixiang Ru 33 Dec 12, 2022
Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral)

DSA^2 F: Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral) This repo is the official imp

如今我已剑指天涯 46 Dec 21, 2022
SE-MSCNN: A Lightweight Multi-scaled Fusion Network for Sleep Apnea Detection Using Single-Lead ECG Signals

SE-MSCNN: A Lightweight Multi-scaled Fusion Network for Sleep Apnea Detection Using Single-Lead ECG Signals Abstract Sleep apnea (SA) is a common slee

null 9 Dec 21, 2022