Behind the Curtain: Learning Occluded Shapes for 3D Object Detection

Related tags

Deep Learning BtcDet
Overview

Behind the Curtain: Learning Occluded Shapes for 3D Object Detection

Acknowledgement

We implement our model, BtcDet, based on [OpenPcdet 0.3.0].

Installation

Requirements

All the codes are tested in the following environment:

Install

b. Install the dependent libraries as follows:

  • Install the dependent python libraries:
pip install -r requirements.txt 
  • Install the SparseConv library, we use the implementation from [spconv].
    • If you use PyTorch 1.1, then make sure you install the spconv v1.0 with (commit 8da6f96) instead of the latest one.
    • If you use PyTorch 1.3+, then you need to install the spconv v1.2. As mentioned by the author of spconv
    git clone -b v1.2.1  https://github.com/traveller59/spconv.git --recursive
    
    cd spconv
    
    sudo apt-get install libboost-all-dev
    
    python setup.py bdist_wheel
    
    cd ./dist 
    
    then use pip to install generated whl file.
    pip install spconv-1.2.1-{your system info}.whl
    
    After that, you should first get out of the spconv directory, then do python import spconv to see if you installed it correctly.

c. Install this btcdet library by running the following command:

cd btcdet
python setup.py develop

Preparation

KITTI Dataset

  • Please download the official KITTI 3D object detection dataset and organize the downloaded files as follows (the road planes could be downloaded from [road plane], which are optional for data augmentation in the training):
BtcDet
├── data
│   ├── kitti
    │   │   │──detection3d  │── ImageSets
                    │   │   │── training
                    │   │   │   ├──calib & velodyne & label_2 & image_2 & (optional: planes) & (optional: depth_2)
                    │   │   │── testing
                    │   │   │   ├──calib & velodyne & image_2
  • Generate the data infos by running the following command:
python -m btcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml

Generate Approximated complete object points:

(at btcdet directory, execute:)

python -m btcdet.datasets.multifindbestfit

Alternatively,

you can use our generated kitti's data including the generated complete object points, download it [here (about 31GBs)] and put the zip file inside data/kitti/ and unzip it as detection3d directory.

Run training:

cd tools/

Single gpu training

mkdir output

mkdir output/kitti_car

python train.py --cfg_file ./cfgs/model_configs/btcdet_kitti_car.yaml --output_dir ../output/kitti_car/ --batch_size 2

Multi gpu training

bash scripts/dist_train.sh 4  --batch_size 8 --gpu_str "0,1,2,3" --cfg_file ./cfgs/model_configs/btcdet_kitti_car.yaml --output_dir ../output/kitti_car/
Comments
  • RuntimeError: CUDA out of memory. Tried to allocate

    RuntimeError: CUDA out of memory. Tried to allocate

    Did anyone face the same issue as below?

    Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/-/BtcDet/btcdet/datasets/multifindbestfit.py", line 494, in find_best_match_boxpnts(all_db_infos_lst, box_dims_lst, sorted_iou, pnt_thresh_best_iou_indices, mirrored_pnts_lst, pnts_lst, coords_num, occ_map, bm_dir_save_path, allrange, nx, ny, voxel_size, max_num_bm=max_num_bm_lst[i], num_extra_coords=num_extra_coords_lst[i], iou_thresh=iou_thresh_lst[i], ex_coords_ratio=ex_coords_ratio_lst[i], nearest_dist=nearest_dist_lst[i], vis=vis, save=save) File "/home/-/BtcDet/btcdet/datasets/multifindbestfit.py", line 330, in find_best_match_boxpnts bm_pnts, bm_coords_num = find_multi_best_match_boxpnts(selected_sorted_iou, cur_box, cur_mirrored_pnts_lst, cur_pnts_lst, selected_mirrored_pnts_lst, selected_pnts_lst, selected_pnt_thresh_best_iou_indices, cur_occ_map, selected_occ_map, max_num_bm=max_num_bm, num_extra_coords=num_extra_coords, iou_thresh=iou_thresh, ex_coords_ratio=ex_coords_ratio, nearest_dist=nearest_dist, vis=vis) File "/home/-/BtcDet/btcdet/datasets/multifindbestfit.py", line 375, in find_multi_best_match_boxpnts mean_instance, min_instance, max_instance = get_batch_stats(dist_l1, box_num_pnts_tensor, box_mask_tensor, box_reversemask_tensor) File "/home/-/BtcDet/btcdet/datasets/multifindbestfit.py", line 351, in get_batch_stats addmax_dist = masked_dist - reversemask_arry RuntimeError: CUDA out of memory. Tried to allocate 198.00 MiB (GPU 0; 10.92 GiB total capacity; 8.94 GiB already allocated; 128.69 MiB free; 9.86 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

    opened by DheerajYarlagadda 3
  • AttributeError: module 'torch' has no attribute 'minimum'

    AttributeError: module 'torch' has no attribute 'minimum'

    Hello!author, i finally succesfully install the spconv, and my environment is torch1.4, when i run the train.py, it raise: File "/data1/hx/BtcDet/btcdet/models/occ_pnt/occ_training_targets/occ_targets_template.py", line 88, in point2coords_inrange coords = torch.minimum(coords, max_grid_tensor) AttributeError: module 'torch' has no attribute 'minimum' why??? is the torch version wrong??(i also use torch1.3, it raise the same wrong)

    opened by pummi823 3
  •  AttributeError: module 'spconv' has no attribute 'SparseConv3d'

    AttributeError: module 'spconv' has no attribute 'SparseConv3d'

    Dear author: thanks for your wonderful work!!! my ubuntu is 20.04 and i follow the "spconv" , which say i can use "pip install spconv-cu113" to install spconv, and after this, i run your "python setup.py develop " and there is no wrong. But when i run the train.py, it raise wrong: AttributeError: module 'spconv' has no attribute 'SparseConv3d', this looks like "python setup.py develop" is actually failed (T T). Am i guess right? could you help me? look forward your response!!!!!!~~~~~

    opened by pummi823 3
  • Try to reproduce performance in paper

    Try to reproduce performance in paper

    Thanks for sharing. This is my reproduction results. The experiment is running on 8 gpus and config is not changed.

    2022-01-10 21:31:31,266 INFO recall_roi_0.5: 0.964236 INFO:btcdet.utils.common_utils:recall_roi_0.5: 0.964236 2022-01-10 21:31:31,266 INFO recall_rcnn_0.5: 0.967569 INFO:btcdet.utils.common_utils:recall_rcnn_0.5: 0.967569 2022-01-10 21:31:31,266 INFO recall_roi_0.7: 0.816458 INFO:btcdet.utils.common_utils:recall_roi_0.7: 0.816458 2022-01-10 21:31:31,267 INFO recall_rcnn_0.7: 0.868125 INFO:btcdet.utils.common_utils:recall_rcnn_0.7: 0.868125 2022-01-10 21:31:31,267 INFO *************************** INFO:btcdet.utils.common_utils: *************************** 2022-01-10 21:31:31,267 INFO precision: 0.531, recall: 0.717, f1: 0.599, precision_factored: 0.536 recall_factored: 0.723, f1_factored: 0.604 INFO:btcdet.utils.common_utils:precision: 0.531, recall: 0.717, f1: 0.599, precision_factored: 0.536 recall_factored: 0.723, f1_factored: 0.604 2022-01-10 21:31:31,267 INFO occ thresh 0.1: 0.953, occ thresh 0.2: 0.953, occ thresh 0.3: 0.953, occ thresh 0.4: 0.938, occ thresh 0.5: 0.914, occ thresh 0.6: 0.883, occ thresh 0.7: 0.833, occ thresh 0.8: 0.720, occ thresh 0.9: 0.233, INFO:btcdet.utils.common_utils:occ thresh 0.1: 0.953, occ thresh 0.2: 0.953, occ thresh 0.3: 0.953, occ thresh 0.4: 0.938, occ thresh 0.5: 0.914, occ thresh 0.6: 0.883, occ thresh 0.7: 0.833, occ thresh 0.8: 0.720, occ thresh 0.9: 0.233, 2022-01-10 21:31:31,267 INFO total_pos_all_portion 0.599 INFO:btcdet.utils.common_utils: total_pos_all_portion 0.599 2022-01-10 21:31:31,269 INFO Average predicted number of objects(3769 samples): 4.237 INFO:btcdet.utils.common_utils:Average predicted number of objects(3769 samples): 4.237 pc, rc (1, 3, 2, 41) (1, 3, 2, 41) pc, rc (1, 3, 2, 41) (1, 3, 2, 41) 2022-01-10 21:31:37,601 INFO Car [email protected], 0.70, 0.70: bbox AP:90.6649, 89.7338, 89.2758 bev AP:90.0933, 88.1567, 87.6482 3d AP:89.0296, 85.5022, 78.6749 aos AP:44.18, 44.36, 44.10 Car [email protected], 0.70, 0.70: bbox AP:96.6044, 95.3285, 92.9333 bev AP:93.5117, 91.5184, 89.2229 3d AP:92.0635, 85.2752, 82.9497 aos AP:45.29, 43.36, 42.14 Car [email protected], 0.50, 0.50: bbox AP:90.6649, 89.7338, 89.2758 bev AP:90.6736, 89.7183, 89.2712 3d AP:90.6736, 89.7061, 89.2479 aos AP:44.18, 44.36, 44.10 Car [email protected], 0.50, 0.50: bbox AP:96.6044, 95.3285, 92.9333 bev AP:96.6282, 95.4054, 92.9708 3d AP:96.6199, 95.3687, 92.9303 aos AP:45.29, 43.36, 42.14

    opened by Eaphan 3
  • About the visualization

    About the visualization

    Hi, Could you tell me which tools you used to visualize the voxel? I really love the style of your figure in the paper, but I can't find a library that allows the transparent visualization of voxels.(Occ in the paper) Thanks for any help.

    opened by Orbis36 2
  • ValueError: need at least one array to concatenate

    ValueError: need at least one array to concatenate

    Getting an error when I run: python -m btcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml

    /home/cad297/anaconda3/envs/btc/lib/python3.6/runpy.py:125: RuntimeWarning: 'btcdet.datasets.kitti.kitti_dataset' found in sys.modules after import of package 'btcdet.datasets.kitti', but prior to execution of 'btcdet.datasets.kitti.kitti_dataset'; this may result in unpredictable behaviour warn(RuntimeWarning(msg)) /home/cad297/projects/BtcDet/btcdet/datasets/kitti/kitti_dataset.py:539: YAMLLoadWarning:calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. dataset_cfg = EasyDict(yaml.load(open(sys.argv[2]))) ROOT_DIR /home/cad297/projects/BtcDet cur_cfg.NAME mask_points_and_boxes_outside_range cur_cfg.NAME shuffle_points cur_cfg.NAME transform_points_to_voxels self.dataset_cfg.INFO_PATH {'train': ['kitti_infos_train.pkl'], 'test': ['kitti_infos_val.pkl']} ---------------Start to generate data infos--------------- cur_cfg.NAME mask_points_and_boxes_outside_range cur_cfg.NAME shuffle_points cur_cfg.NAME transform_points_to_voxels Traceback (most recent call last): File "/home/cad297/anaconda3/envs/btc/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/cad297/anaconda3/envs/btc/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/cad297/projects/BtcDet/btcdet/datasets/kitti/kitti_dataset.py", line 548, in save_path=path File "/home/cad297/projects/BtcDet/btcdet/datasets/kitti/kitti_dataset.py", line 475, in create_kitti_infos kitti_infos_train = dataset.get_infos(num_workers=workers, has_label=True, count_inside_pts=True) File "/home/cad297/projects/BtcDet/btcdet/datasets/kitti/kitti_dataset.py", line 201, in get_infos return list(infos) File "/home/cad297/anaconda3/envs/btc/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/home/cad297/anaconda3/envs/btc/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/home/cad297/anaconda3/envs/btc/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/home/cad297/anaconda3/envs/btc/lib/python3.6/concurrent/futures/thread.py", line56, in run result = self.fn(*self.args, **self.kwargs) File "/home/cad297/projects/BtcDet/btcdet/datasets/kitti/kitti_dataset.py", line 158, in process_single_scene annotations['bbox'] = np.concatenate([obj.box2d.reshape(1, 4) for obj in obj_list], axis=0) File "<array_function internals>", line 6, in concatenate ValueError: need at least one array to concatenate

    Anyone seen the same? Or have been able to fix?

    opened by cdiazruiz 1
  • Could you provide the final checkpoint file related to the paper?

    Could you provide the final checkpoint file related to the paper?

    Thank you for your sharing, that's amazing job! But I can not reimplement the precision result in your paper, could you give me a favor to provide relative checkpoint file?

    opened by hunterkeith2017 1
  • ModuleNotFoundError: No module named 'torch' Error when trying to run setup.py

    ModuleNotFoundError: No module named 'torch' Error when trying to run setup.py

    Hi, I've done every step before this and they all works.

    When I try to run sudo python3 setup.py develop it returns no module named torch error but torch is installed and it works. How can I fix this problem.

    Thanks in advance

    opened by Sekomer 1
  • Question bout min_instance or max_instance in heuristic

    Question bout min_instance or max_instance in heuristic

    Hi @Xharlie Thanks for the released code. I have a question about this: # min_instance = min_instance.view(gt_boxnum, candidate_num) max_instance = max_instance.view(gt_boxnum, candidate_num) and heuristic = max_instance + ex_coords_ratio / extra_coord_nums.unsqueeze(0) + (sorted_iou.unsqueeze(0) < iou_thresh) * 2.0 + (extra_coord_nums.unsqueeze(0) < 30) * 1.0

    why not use min_instance? I think min_instance is more reasonable

    opened by Joeless 0
  • FileNotFoundError:No such file or directory: '/home/z/BtcDet-main/data/kitti/detection3d/gt_database/00003_car_0.bin

    FileNotFoundError:No such file or directory: '/home/z/BtcDet-main/data/kitti/detection3d/gt_database/00003_car_0.bin

    When you run this statement ,python -m btcdet.datasets.multifindbestfit , the following error occurs FileNotFoundError: No such file or directory: '/home/z/BtcDet-main/data/kitti/detection3d/gt_database/00003_car_0.bin

    opened by 123jyw 0
  • how to create directory 'bm_50maxdist_2num_car'

    how to create directory 'bm_50maxdist_2num_car'

    hello

    i tried to train btcdet. but error occured.

    error message : FileNotFoundError: [Errno 2] No such file or directory: '/tmp/BtcDet/data/kitti/detection3d/bm_50maxdist_2num_car/1788_0.pkl'

    do you know how to create directory and *.pkl files?

    opened by yeoeun-oh 2
  • ModuleNotFoundError: No module named 'btcdet.utils'

    ModuleNotFoundError: No module named 'btcdet.utils'

    after 'python3 setup.py develop'

    Traceback (most recent call last): File "train.py", line 6, in from test import repeat_eval_ckpt File "/tmp/BtcDet/tools/test.py", line 14, in from btcdet.datasets import build_dataloader File "/usr/local/lib/python3.6/dist-packages/btcdet/datasets/init.py", line 5, in from btcdet.utils import common_utils ModuleNotFoundError: No module named 'btcdet.utils'

    opened by yeoeun-oh 1
  • Missing Configuration Files: Pedestrians and Cyclists

    Missing Configuration Files: Pedestrians and Cyclists

    Hi,

    I have been working on the repository and have reproduced the results for "Car" using "btcdet_kitti_car.yaml". However, since there is no config files available for either "Pedestrians" or "Cyclist", I would request you to please provide for the config files for those classes.

    Thank you!

    opened by adi018 2
  • TypeError: list indices must be integers or slices, not str

    TypeError: list indices must be integers or slices, not str

    It seems that this repo has many bugs, such as this dbinfos, there is no 'kitti_dbinfos_val.pkl' under the kitti dataset. So this error will be reported, has anyone encountered the same error?

      File "/home/algo-3/work/BtcDet/btcdet/datasets/multifindbestfit.py", line 38, in extract_allpnts
        all_db_infos = pickle.load(f)[type]
    TypeError: list indices must be integers or slices, not str
    

    https://github.com/Xharlie/BtcDet/blob/80bee34f2f40931600f812a6edbcb27e51cb7ec3/btcdet/datasets/multifindbestfit.py#L33

    opened by Leozyc-waseda 0
  • RuntimeError:expected backend CUDA and dtype Float but got backend CUDA and dtype Long

    RuntimeError:expected backend CUDA and dtype Float but got backend CUDA and dtype Long

    When you run this statement ,python -m btcdet.datasets.multifindbestfit , the following error occursRuntimeError:expected backend CUDA and dtype Float but got backend CUDA and dtype Long.

    opened by 123jyw 0
Owner
Qiangeng Xu
Qiangeng Xu
Code for CMaskTrack R-CNN (proposed in Occluded Video Instance Segmentation)

CMaskTrack R-CNN for OVIS This repo serves as the official code release of the CMaskTrack R-CNN model on the Occluded Video Instance Segmentation data

Q . J . Y 61 Nov 25, 2022
Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

HamasKhan 3 Jul 8, 2022
3D AffordanceNet is a 3D point cloud benchmark consisting of 23k shapes from 23 semantic object categories, annotated with 56k affordance annotations and covering 18 visual affordance categories.

3D AffordanceNet This repository is the official experiment implementation of 3D AffordanceNet benchmark. 3D AffordanceNet is a 3D point cloud benchma

null 49 Dec 1, 2022
Neural Magic Eye: Learning to See and Understand the Scene Behind an Autostereogram, arXiv:2012.15692.

Neural Magic Eye Preprint | Project Page | Colab Runtime Official PyTorch implementation of the preprint paper "NeuralMagicEye: Learning to See and Un

Zhengxia Zou 56 Jul 15, 2022
Learning Intents behind Interactions with Knowledge Graph for Recommendation, WWW2021

Learning Intents behind Interactions with Knowledge Graph for Recommendation This is our PyTorch implementation for the paper: Xiang Wang, Tinglin Hua

null 158 Dec 15, 2022
Learning Skeletal Articulations with Neural Blend Shapes

This repository provides an end-to-end library for automatic character rigging and blend shapes generation as well as a visualization tool. It is based on our work Learning Skeletal Articulations with Neural Blend Shapes that is published in SIGGRAPH 2021.

Peizhuo 504 Dec 30, 2022
Code for Learning Manifold Patch-Based Representations of Man-Made Shapes, in ICLR 2021.

LearningPatches | Webpage | Paper | Video Learning Manifold Patch-Based Representations of Man-Made Shapes Dmitriy Smirnov, Mikhail Bessmeltsev, Justi

Dima Smirnov 22 Nov 14, 2022
SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images

SymmetryNet SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images ACM Transactions on Gra

null 26 Dec 5, 2022
Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

null 5 Dec 10, 2022
Yolo object detection - Yolo object detection with python

How to run download required files make build_image make download Docker versio

null 3 Jan 26, 2022
Official implementation of NPMs: Neural Parametric Models for 3D Deformable Shapes - ICCV 2021

NPMs: Neural Parametric Models Project Page | Paper | ArXiv | Video NPMs: Neural Parametric Models for 3D Deformable Shapes Pablo Palafox, Aljaz Bozic

PabloPalafox 109 Nov 22, 2022
This repository contains the accompanying code for Deep Virtual Markers for Articulated 3D Shapes, ICCV'21

Deep Virtual Markers This repository contains the accompanying code for Deep Virtual Markers for Articulated 3D Shapes, ICCV'21 Getting Started Get sa

KimHyomin 45 Oct 7, 2022
The official implementation of the research paper "DAG Amendment for Inverse Control of Parametric Shapes"

DAG Amendment for Inverse Control of Parametric Shapes This repository is the official Blender implementation of the paper "DAG Amendment for Inverse

Elie Michel 157 Dec 26, 2022
Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

null 235 Dec 26, 2022
This is the official source code for SLATE. We provide the code for the model, the training code, and a dataset loader for the 3D Shapes dataset. This code is implemented in Pytorch.

SLATE This is the official source code for SLATE. We provide the code for the model, the training code and a dataset loader for the 3D Shapes dataset.

Gautam Singh 66 Dec 26, 2022
Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera.

Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera. This project prepares training and testing data for various deep learning projects such as 6D object pose estimation projects singleshotpose, as well as object detection and instance segmentation projects.

null 305 Dec 16, 2022
Official PyTorch implementation of Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

This is the official PyTorch implementation of our paper: "Joint Object Detection and Multi-Object Tracking with Graph Neural Networks". Our project website and video demos are here.

Richard Wang 443 Dec 6, 2022
CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection

CLOCs is a novel Camera-LiDAR Object Candidates fusion network. It provides a low-complexity multi-modal fusion framework that improves the performance of single-modality detectors. CLOCs operates on the combined output candidates of any 3D and any 2D detector, and is trained to produce more accurate 3D and 2D detection results.

Su Pang 254 Dec 16, 2022