Code for CVPR 2022 paper "SoftGroup for Instance Segmentation on 3D Point Clouds"

Overview

SoftGroup

PWC PWC Architecture

We provide code for reproducing results of the paper SoftGroup for 3D Instance Segmentation on Point Clouds (CVPR 2022)

Author: Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, and Chang D. Yoo.

Introduction

Existing state-of-the-art 3D instance segmentation methods perform semantic segmentation followed by grouping. The hard predictions are made when performing semantic segmentation such that each point is associated with a single class. However, the errors stemming from hard decision propagate into grouping that results in (1) low overlaps between the predicted instance with the ground truth and (2) substantial false positives. To address the aforementioned problems, this paper proposes a 3D instance segmentation method referred to as SoftGroup by performing bottom-up soft grouping followed by top-down refinement. SoftGroup allows each point to be associated with multiple classes to mitigate the problems stemming from semantic prediction errors and suppresses false positive instances by learning to categorize them as background. Experimental results on different datasets and multiple evaluation metrics demonstrate the efficacy of SoftGroup. Its performance surpasses the strongest prior method by a significant margin of +6.2% on the ScanNet v2 hidden test set and +6.8% on S3DIS Area 5 of AP_50.

Learderboard

Feature

  • State of the art performance on the ScanNet benchmark and S3DIS dataset (3/Mar/2022).
  • High speed of 345 ms per scan on ScanNet dataset, which is comparable with the existing fastest methods (HAIS).
  • Reproducibility code for both ScanNet and S3DIS datasets.

Installation

Please refer to installation guide.

Data Preparation

Please refer to data preparation for preparing the S3DIS and ScanNet v2 dataset.

Pretrained models

Dataset AP AP_50 AP_25 Download
S3DIS 51.4 66.5 75.4 model
ScanNet v2 46.0 67.6 78.9 model

Training

We use the checkpoint of HAIS as pretrained backbone. Download the pretrained HAIS model at here at put it in SoftGroup/ directory.

Training S3DIS dataset

First, finetune the pretrained HAIS point-wise prediction network (backbone) on S3DIS.

python train.py --config config/softgroup_fold5_backbone_s3dis.yaml

Then, train model from frozen backbone.

python train.py --config config/softgroup_fold5_default_s3dis.yaml

Training ScanNet V2 dataset

Training on ScanNet doesnot require finetuning the backbone. Just freeze pretrained backbone and train the model.

python train.py --config config/softgroup_default_scannet.yaml

Inference

Testing for S3DIS dataset.

CUDA_VISIBLE_DEVICES=0 python test_s3dis.py --config config/softgroup_fold5_phase2_s3dis.yaml --pretrain $PATH_TO_PRETRAIN_MODEL$

Testing for ScanNet V2 dataset.

CUDA_VISIBLE_DEVICES=0 python test.py --config config/softgroup_default_scannet.yaml --pretrain $PATH_TO_PRETRAIN_MODEL$

Visualization

We provide visualization tools based on Open3D (tested on Open3D 0.8.0).

pip install open3D==0.8.0
python visualize_open3d.py --data_path {} --prediction_path {} --data_split {} --room_name {} --task {}

Please refer to visualize_open3d.py for more details.

Citation

If you find our work helpful for your research. Please consider citing our paper.

@inproceedings{vu2022softgroup,
  title={SoftGroup for 3D Instance Segmentation on 3D Point Clouds},
  author={Vu, Thang and Kim, Kookhoi and Luu, Tung M. and Nguyen, Xuan Thanh and Yoo, Chang D.},
  booktitle={CVPR},
  year={2022}
}
Comments
  • about S3DIS test

    about S3DIS test

    Hi!Thanks for the great work I'm running the previous version of the program.I can test successfully according to readme.md. first , Generate preprocessed data after running bash prepare_data.sh; second, I use your open source pre training model(softgroup_s3dis.pth)to test and test data is Area2. then run "python test_s3dis.py --config/softgroup_fold5_default_s3dis.yaml'', generate test results include semantic_pred、instance_pred; finally, I use script "visualization.py" for visualization。But, Inference result is much worse than ground true,I don't know where went wrong。I hope I can get your answer。

    opened by CodeFly-123 35
  • Spconv and Cuda error while training on my own dataset

    Spconv and Cuda error while training on my own dataset

    Hi, I got the following error when training on my own dataset. However, I could reproduce the results on S3DIS dataset without any error. Can you please look into my error and give me suggestion what might have been wrong? My dataset is same like S3DIS dataset with xyzrgb format. I have the following configuration: CUDA = V10.2.89 torch = 1.11.0 spconv-cu102 = 2.1.21 Ubuntu 18.04 I am really sorry for a long error message. Thank you for your help !

    (softgroup2) shrijan_pf@BQ-DX1100-CT2:~/SoftGroup$ ./tools/dist_train.sh configs/softgroup_s3dis_backbone_fold5_mintA.yaml 1 2022-04-28 13:43:26,027 - INFO - Config: model: channels: 32 num_blocks: 7 semantic_classes: 3 #changed, original 13 instance_classes: 3 #changed, original 13 sem2ins_classes: [0, 1] semantic_only: True ignore_label: -100 grouping_cfg: score_thr: 0.2 radius: 0.04 mean_active: 300 class_numpoint_mean: [1823, 7457, 6189, 7424, 34229, 1724, 5439, 6016, 39796, 5279, 5092, 12210, 10225] npoint_thr: 0.05 # absolute if class_numpoint == -1, relative if class_numpoint != -1 ignore_classes: [-99] instance_voxel_cfg: scale: 50 spatial_shape: 20 train_cfg: max_proposal_num: 200 pos_iou_thr: 0.5 test_cfg: x4_split: True cls_score_thr: 0.001 mask_score_thr: -0.5 min_npoint: 100 fixed_modules: []

    data: train: type: 's3dis' data_root: 'dataset/s3dis/preprocess_mint_sample' #changed original 'dataset/s3dis/preprocess' prefix: ['Area_2'] # changed original ['Area_1', 'Area_2', 'Area_3', 'Area_4', 'Area_6'] suffix: '_inst_nostuff.pth' repeat: 20 training: True voxel_cfg: scale: 50 spatial_shape: [128, 512] max_npoint: 2500000 min_npoint: 5000 test: type: 's3dis' data_root: 'dataset/s3dis/preprocess_mint_sample' #changed original 'dataset/s3dis/preprocess' prefix: 'Area_1' #changed, original 'Area_5' suffix: '_inst_nostuff.pth' training: False voxel_cfg: scale: 50 spatial_shape: [128, 512] max_npoint: 2500000 min_npoint: 5000

    dataloader: train: batch_size: 1 #changed;original was 4 num_workers: 4 test: batch_size: 1 num_workers: 1

    optimizer: type: 'Adam' lr: 0.004

    save_cfg: semantic: True offset: True instance: False

    fp16: False epochs: 20 step_epoch: 0 save_freq: 2 pretrain: './hais_ckpt_spconv2.pth' work_dir: ''

    2022-04-28 13:43:26,027 - INFO - Distributed: True 2022-04-28 13:43:26,027 - INFO - Mix precision training: False 2022-04-28 13:43:27,398 - INFO - Load train dataset: 580 scans 2022-04-28 13:43:27,399 - INFO - Load test dataset: 32 scans 2022-04-28 13:43:27,399 - INFO - Load pretrain from ./hais_ckpt_spconv2.pth 2022-04-28 13:43:27,629 - INFO - removed keys in source state_dict due to size mismatch: semantic_linear.3.weight, semantic_linear.3.bias 2022-04-28 13:43:27,629 - INFO - missing keys in source state_dict: semantic_linear.3.weight, semantic_linear.3.bias 2022-04-28 13:43:27,629 - INFO - unexpected key in source state_dict: tiny_unet.blocks.block0.conv_branch.0.weight, tiny_unet.blocks.block0.conv_branch.0.bias, tiny_unet.blocks.block0.conv_branch.0.running_mean, tiny_unet.blocks.block0.conv_branch.0.running_var, tiny_unet.blocks.block0.conv_branch.0.num_batches_tracked, tiny_unet.blocks.block0.conv_branch.2.weight, tiny_unet.blocks.block0.conv_branch.3.weight, tiny_unet.blocks.block0.conv_branch.3.bias, tiny_unet.blocks.block0.conv_branch.3.running_mean, tiny_unet.blocks.block0.conv_branch.3.running_var, tiny_unet.blocks.block0.conv_branch.3.num_batches_tracked, tiny_unet.blocks.block0.conv_branch.5.weight, tiny_unet.blocks.block1.conv_branch.0.weight, tiny_unet.blocks.block1.conv_branch.0.bias, tiny_unet.blocks.block1.conv_branch.0.running_mean, tiny_unet.blocks.block1.conv_branch.0.running_var, tiny_unet.blocks.block1.conv_branch.0.num_batches_tracked, tiny_unet.blocks.block1.conv_branch.2.weight, tiny_unet.blocks.block1.conv_branch.3.weight, tiny_unet.blocks.block1.conv_branch.3.bias, tiny_unet.blocks.block1.conv_branch.3.running_mean, tiny_unet.blocks.block1.conv_branch.3.running_var, tiny_unet.blocks.block1.conv_branch.3.num_batches_tracked, tiny_unet.blocks.block1.conv_branch.5.weight, tiny_unet.conv.0.weight, tiny_unet.conv.0.bias, tiny_unet.conv.0.running_mean, tiny_unet.conv.0.running_var, tiny_unet.conv.0.num_batches_tracked, tiny_unet.conv.2.weight, tiny_unet.u.blocks.block0.conv_branch.0.weight, tiny_unet.u.blocks.block0.conv_branch.0.bias, tiny_unet.u.blocks.block0.conv_branch.0.running_mean, tiny_unet.u.blocks.block0.conv_branch.0.running_var, tiny_unet.u.blocks.block0.conv_branch.0.num_batches_tracked, tiny_unet.u.blocks.block0.conv_branch.2.weight, tiny_unet.u.blocks.block0.conv_branch.3.weight, tiny_unet.u.blocks.block0.conv_branch.3.bias, tiny_unet.u.blocks.block0.conv_branch.3.running_mean, tiny_unet.u.blocks.block0.conv_branch.3.running_var, tiny_unet.u.blocks.block0.conv_branch.3.num_batches_tracked, tiny_unet.u.blocks.block0.conv_branch.5.weight, tiny_unet.u.blocks.block1.conv_branch.0.weight, tiny_unet.u.blocks.block1.conv_branch.0.bias, tiny_unet.u.blocks.block1.conv_branch.0.running_mean, tiny_unet.u.blocks.block1.conv_branch.0.running_var, tiny_unet.u.blocks.block1.conv_branch.0.num_batches_tracked, tiny_unet.u.blocks.block1.conv_branch.2.weight, tiny_unet.u.blocks.block1.conv_branch.3.weight, tiny_unet.u.blocks.block1.conv_branch.3.bias, tiny_unet.u.blocks.block1.conv_branch.3.running_mean, tiny_unet.u.blocks.block1.conv_branch.3.running_var, tiny_unet.u.blocks.block1.conv_branch.3.num_batches_tracked, tiny_unet.u.blocks.block1.conv_branch.5.weight, tiny_unet.deconv.0.weight, tiny_unet.deconv.0.bias, tiny_unet.deconv.0.running_mean, tiny_unet.deconv.0.running_var, tiny_unet.deconv.0.num_batches_tracked, tiny_unet.deconv.2.weight, tiny_unet.blocks_tail.block0.i_branch.0.weight, tiny_unet.blocks_tail.block0.conv_branch.0.weight, tiny_unet.blocks_tail.block0.conv_branch.0.bias, tiny_unet.blocks_tail.block0.conv_branch.0.running_mean, tiny_unet.blocks_tail.block0.conv_branch.0.running_var, tiny_unet.blocks_tail.block0.conv_branch.0.num_batches_tracked, tiny_unet.blocks_tail.block0.conv_branch.2.weight, tiny_unet.blocks_tail.block0.conv_branch.3.weight, tiny_unet.blocks_tail.block0.conv_branch.3.bias, tiny_unet.blocks_tail.block0.conv_branch.3.running_mean, tiny_unet.blocks_tail.block0.conv_branch.3.running_var, tiny_unet.blocks_tail.block0.conv_branch.3.num_batches_tracked, tiny_unet.blocks_tail.block0.conv_branch.5.weight, tiny_unet.blocks_tail.block1.conv_branch.0.weight, tiny_unet.blocks_tail.block1.conv_branch.0.bias, tiny_unet.blocks_tail.block1.conv_branch.0.running_mean, tiny_unet.blocks_tail.block1.conv_branch.0.running_var, tiny_unet.blocks_tail.block1.conv_branch.0.num_batches_tracked, tiny_unet.blocks_tail.block1.conv_branch.2.weight, tiny_unet.blocks_tail.block1.conv_branch.3.weight, tiny_unet.blocks_tail.block1.conv_branch.3.bias, tiny_unet.blocks_tail.block1.conv_branch.3.running_mean, tiny_unet.blocks_tail.block1.conv_branch.3.running_var, tiny_unet.blocks_tail.block1.conv_branch.3.num_batches_tracked, tiny_unet.blocks_tail.block1.conv_branch.5.weight, tiny_unet_outputlayer.0.weight, tiny_unet_outputlayer.0.bias, tiny_unet_outputlayer.0.running_mean, tiny_unet_outputlayer.0.running_var, tiny_unet_outputlayer.0.num_batches_tracked, iou_score_linear.weight, iou_score_linear.bias, mask_linear.0.weight, mask_linear.0.bias, mask_linear.2.weight, mask_linear.2.bias 2022-04-28 13:43:27,630 - INFO - Training [Exception|implicit_gemm]feat=torch.Size([17843, 64]),w=torch.Size([32, 2, 2, 2, 64]),pair=torch.Size([8, 17848]),act=17848,issubm=False,istrain=True SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue. Traceback (most recent call last): File "./tools/train.py", line 185, in main() File "./tools/train.py", line 178, in main train(epoch, model, optimizer, scaler, train_loader, cfg, logger, writer) File "./tools/train.py", line 48, in train loss, log_vars = model(batch, return_loss=True) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 963, in forward output = self.module(*inputs[0], **kwargs[0]) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shrijan_pf/SoftGroup/softgroup/model/softgroup.py", line 97, in forward return self.forward_train(**batch) File "/home/shrijan_pf/SoftGroup/softgroup/util/utils.py", line 171, in wrapper return func(*new_args, **new_kwargs) File "/home/shrijan_pf/SoftGroup/softgroup/model/softgroup.py", line 109, in forward_train semantic_scores, pt_offsets, output_feats = self.forward_backbone(input, v2p_map) File "/home/shrijan_pf/SoftGroup/softgroup/model/softgroup.py", line 263, in forward_backbone output = self.unet(output) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shrijan_pf/SoftGroup/softgroup/model/blocks.py", line 139, in forward output_decoder = self.deconv(output_decoder) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/spconv/pytorch/modules.py", line 137, in forward input = module(input) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/spconv/pytorch/conv.py", line 446, in forward input._timer, self.fp32_accum) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/cuda/amp/autocast_mode.py", line 118, in decorate_fwd return fwd(*args, kwargs) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/spconv/pytorch/functional.py", line 200, in forward raise e File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/spconv/pytorch/functional.py", line 191, in forward fp32_accum) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/spconv/pytorch/ops.py", line 1118, in implicit_gemm fp32_accum=fp32_accum) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/spconv/algo.py", line 661, in tune_and_cache GemmMainUnitTest.stream_synchronize(stream) RuntimeError: /io/build/temp.linux-x86_64-3.7/spconv/build/src/cumm/gemm/main/GemmMainUnitTest/GemmMainUnitTest_stream_synchronize.cc(11) CUDA error 700 terminate called after throwing an instance of 'c10::CUDAError' what(): CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1646755861072/work/c10/cuda/CUDACachingAllocator.cpp:1230 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7f5183c301bd in /home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: + 0x1ec97 (0x7f5183e9ac97 in /home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void) + 0x23a (0x7f5183e9f1fa in /home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) frame #3: + 0x2edaf8 (0x7f51ca243af8 in /home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/lib/libtorch_python.so) frame #4: c10::TensorImpl::release_resources() + 0x175 (0x7f5183c16fb5 in /home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/lib/libc10.so) frame #5: + 0x1daa99 (0x7f51ca130a99 in /home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/lib/libtorch_python.so) frame #6: + 0x4cf71c (0x7f51ca42571c in /home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/lib/libtorch_python.so) frame #7: THPVariable_subclass_dealloc(_object) + 0x299 (0x7f51ca425a39 in /home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/lib/libtorch_python.so) frame #8: + 0xfc359 (0x55c9017e7359 in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #9: + 0xfac88 (0x55c9017e5c88 in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #10: + 0xfac88 (0x55c9017e5c88 in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #11: + 0xfa6a8 (0x55c9017e56a8 in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #12: + 0xfb128 (0x55c9017e6128 in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #13: + 0xfb13c (0x55c9017e613c in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #14: + 0xfb13c (0x55c9017e613c in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #15: + 0xfb13c (0x55c9017e613c in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #16: + 0xfb13c (0x55c9017e613c in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #17: + 0x12b4a7 (0x55c9018164a7 in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #18: PyDict_SetItemString + 0x89 (0x55c901822a19 in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #19: PyImport_Cleanup + 0xab (0x55c901897b8b in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #20: Py_FinalizeEx + 0x64 (0x55c90190c714 in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #21: + 0x232e20 (0x55c90191de20 in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #22: _Py_UnixMain + 0x3c (0x55c90191e18c in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python) frame #23: __libc_start_main + 0xe7 (0x7f51e65b4c87 in /lib/x86_64-linux-gnu/libc.so.6) frame #24: + 0x1d803a (0x55c9018c303a in /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python)

    ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 13703) of binary: /home/shrijan_pf/anaconda3/envs/softgroup2/bin/python Traceback (most recent call last): File "/home/shrijan_pf/anaconda3/envs/softgroup2/bin/torchrun", line 33, in sys.exit(load_entry_point('torch==1.11.0', 'console_scripts', 'torchrun')()) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 345, in wrapper return f(*args, **kwargs) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/distributed/run.py", line 724, in main run(args) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/distributed/run.py", line 718, in run )(*cmd_args) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/home/shrijan_pf/anaconda3/envs/softgroup2/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent failures=result.failures, torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

    ./tools/train.py FAILED

    Failures: <NO_OTHER_FAILURES>

    Root Cause (first observed failure): [0]: time : 2022-04-28_13:43:50 host : BQ-DX1100-CT2 rank : 0 (local_rank: 0) exitcode : -6 (pid: 13703) error_file: <N/A> traceback : Signal 6 (SIGABRT) received by PID 13703

    opened by SijanNeupane49 19
  • Memory leak?

    Memory leak?

    When I am training on batch size 4 and numworkers 4 my training starts off at ~8gb. During the epochs it climbs up with steps of ~4gb. Eventually Ubuntu crashes, maybe because of another issue but the growing memory in the epoch is quite strange. I have not changed anything in the training or the softgroup model, is anyone aware of this issue and how can I fix the growing memory. I tried num workers 0 and increasing batch size but it keeps crashing, even in the first epoch compared to every ~ 4th epoch with batch 4 and num workers 4.

    opened by GooseFather990 17
  • can I use the model on windows system

    can I use the model on windows system

    Hi,

    Thanks for the open source code. As the title said, can I use the implementation on Windows? I see the installation instruction is binding to Linux system.

    Many thanks, Enoch

    opened by EnochYing 15
  • dataset

    dataset

    Hi, thank you for your job! I made a dateset like S3DIS, but is smaller smaller than it. preprocess use --prepare_data.sh. But when I train, …… WRONG: point num < 20000, continue I try to change 20000 to 2000 or 20 , but it also happen. can you give me some advice when I use small data. THANK YOU!

    opened by Atopis 15
  • Custom data set

    Custom data set

    I'm trying to train the model with a completely different dataset. I'm struggling to set the values in the config file, is there some documentation about setting them correctly or their meaning/influence?

    opened by IgnacioSan22 9
  • Incomplete categories after visualisation

    Incomplete categories after visualisation

    Hi, when running visualize_open3d.py the parameter about the --prediction_path is passed in. /exp/scannetv2/softgroup/softgroup_default_scannet/result? E978E729BA8E1E361C3A38EF052FF2F0 Is this the result of scene0016_00?

    opened by Naruto-xsl 8
  • RuntimeError: expected a non-empty list of Tensors

    RuntimeError: expected a non-empty list of Tensors

    Hello,when I run python train.py --config config/softgroup_fold5_default_s3dis.yaml,it throw the following error.Could you please tell me how to deal it?

    Traceback (most recent call last): File "test_s3dis.py", line 338, in test(model, model_fn, data_name, cfg.test_epoch) File "test_s3dis.py", line 66, in test preds = model_fn(batch, model, epoch) File "/home/vipuser/PycharmProjects/pythonProject/SoftGroup_main1/model/softgroup/softgroup.py", line 482, in test_model_fn ret = model(input_, p2v_map, coords_float, batch_idxs, batch_offsets, epoch, 'test', split=True, semantic_only=semantic_only) File "/home/vipuser/miniconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/vipuser/PycharmProjects/pythonProject/SoftGroup_main1/model/softgroup/softgroup.py", line 385, in forward proposals_idx = torch.cat(proposals_idx_list, dim=0) RuntimeError: expected a non-empty list of Tensors

    opened by PengboLi1998 7
  • Dataloader worker is killed

    Dataloader worker is killed

    Hi, I encountered a suspicious problem after running a few hundreds of training epochs:

    RuntimeError: DataLoader worker (pid 75890) is killed by signal: Killed.
    

    The dataloader worker seemed to be stuck. Have you met the same problem and how can I solve it?

    opened by cshizhe 7
  • Is it possible to train SoftGroup on our own dataset?

    Is it possible to train SoftGroup on our own dataset?

    I have a dataset a point cloud. I labeled it using CloudCompare and add two Scalar Field values: the first is the semantic class and the second is the instance class. Could I train it on my dataset?

    opened by bilel-bj 7
  • Inference with Scannet validation and test dataset

    Inference with Scannet validation and test dataset

    Hello, first of all, congratulation on your fantastic work. Currently, I have the code working. First, I prepared the dataset running the scripts:

    • python split_data.py
    • python prepare_data_inst.py --data_split val
    • python prepare_data_inst.py --data_split test Once I had the dataset in an appropriated order, when I ran the latest version (https://github.com/thangvubk/SoftGroup/commit/cf88d9be41ae83a70f9100856a3ca15ee4ddcee9) of the inference (test.py) with the recommended configuration, I obtain semantic_prediction equal to 1 (in every point) on each point and the pred_instances is empty (in both cases: validation set using labels and test set without labeling). As an extra information, the only part which is different in my environment is the Pytorch version, which is 1.10.2. How can I solve this problem?
    opened by claragarciamoll 6
  • Ballquery_batch_p’s meaning

    Ballquery_batch_p’s meaning

    hello,author. There is such a function called 'ops.ballquery_batch_p'. May I ask what does it mean?The output is a number ( 66365386): nActive = ops.ballquery_batch_p(coords, batch_idxs, batch_offsets, idx, start_len, n, meanActive=300, radius=0.04)
    if nActive <= n * meanActive(70329600): break meanActive = int(nActive // n + 1) idx = idx[:nActive] # (66365386): [10618,10632,...130336,230608] return idx, start_len

    opened by xiyufeng2 2
  • Same training time for SoftGroup and SoftGroup++

    Same training time for SoftGroup and SoftGroup++

    Hi! I trained SoftGroup/SoftGroup++ from scratch on STPLS3D, using configs: softgroup++_stpls3d.yaml and softgroup_stpls3d.yaml respectively. However, the training time in both cases is the same ~ 6 hours. Does SoftGroup also use Octree k-NN independent from "with_octree" config?

    opened by Ritchizh 0
  • Ignore_label

    Ignore_label

    Hi,

    Im training SoftGroup on my own dataset. I wondered whether ignore_label regards semantic labels or instance labels or both? I have labelled all points in my dataset that don't belong to any instance with -1 instance label. I therefore set ignore_label to -1.

    Training the backbone on semantic segmentation worked fine, however, training the instance segmentation fails because of dimension errors:

    /aten/src/ATen/native/cuda/Loss.cu:257: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [8,0,0] Assertion t >= 0 && t < n_classes failed and CUDA error: device-side assert triggered

    The error happens in instance_loss() calculation, more specifically line 232, in instance_loss cls_loss = F.cross_entropy(cls_scores, labels).

    Could it be that the model does not ignore the -1 instances and therefore there is a dimension error?

    Thank you in advance for any answers!

    opened by Augustab 0
  • Softgroup++ hasn't implemented semantic kitti config?

    Softgroup++ hasn't implemented semantic kitti config?

    It seems like kitti config only existed in softgroup's config directory.

    Do you have any plan for implementation of Softgroup++'s semantic kitti support?

    Thanks

    opened by gunjupark 0
  • About Instance Segmentation

    About Instance Segmentation

    Hello! Thank you for your awesome work for instance segmentation. I notice that certain individual point may has multiple instance label by using Softgroup. PointGroup define the Instance segmentation task as finding the foreground objects in a scene and mark each object instance with a unique label. So, is multiple instance label appropriate?

    opened by aoligei140 0
  • RuntimeError: CUDA error: an illegal memory access was encountered

    RuntimeError: CUDA error: an illegal memory access was encountered

    When i train the model on ScannetV2, i confront the problem during validation "RuntimeError: CUDA error: an illegal memory access was encountered terminate called after throwing an instance of 'c10::CUDAError' what(): CUDA error: an illegal memory access was encountered"

    So, is it concerned about the memory size? and if i skip validation, how can i deal with the problem during test ?

    opened by YuerGu 0
Owner
Thang Vu
My research involves in Deep Learning for Computer Vision (image enhancement, object detection, segmentation) and other AI related fields.
Thang Vu
Code for CVPR 2022 paper "Bailando: 3D dance generation via Actor-Critic GPT with Choreographic Memory"

Bailando Code for CVPR 2022 (oral) paper "Bailando: 3D dance generation via Actor-Critic GPT with Choreographic Memory" [Paper] | [Project Page] | [Vi

Li Siyao 237 Dec 29, 2022
Official code for "Bridging Video-text Retrieval with Multiple Choice Questions", CVPR 2022 (Oral).

Bridging Video-text Retrieval with Multiple Choice Questions, CVPR 2022 (Oral) Paper | Project Page | Pre-trained Model | CLIP-Initialized Pre-trained

Applied Research Center (ARC), Tencent PCG 99 Jan 6, 2023
Official code for ROCA: Robust CAD Model Retrieval and Alignment from a Single Image (CVPR 2022)

ROCA: Robust CAD Model Alignment and Retrieval from a Single Image (CVPR 2022) Code release of our paper ROCA. Check out our video, paper, and website

null 123 Dec 25, 2022
Source code of our TPAMI'21 paper Dual Encoding for Video Retrieval by Text and CVPR'19 paper Dual Encoding for Zero-Example Video Retrieval.

Dual Encoding for Video Retrieval by Text Source code of our TPAMI'21 paper Dual Encoding for Video Retrieval by Text and CVPR'19 paper Dual Encoding

null 81 Dec 1, 2022
Code for the head detector (HeadHunter) proposed in our CVPR 2021 paper Tracking Pedestrian Heads in Dense Crowd.

Head Detector Code for the head detector (HeadHunter) proposed in our CVPR 2021 paper Tracking Pedestrian Heads in Dense Crowd. The head_detection mod

Ramana Subramanyam 76 Dec 6, 2022
WACV 2022 Paper - Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching

Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching Code based on our WACV 2022 Accepted Paper: https://arxiv.org/pdf/

Andres 13 Dec 17, 2022
CVPR 2021 Oral paper "LED2-Net: Monocular 360˚ Layout Estimation via Differentiable Depth Rendering" official PyTorch implementation.

LED2-Net This is PyTorch implementation of our CVPR 2021 Oral paper "LED2-Net: Monocular 360˚ Layout Estimation via Differentiable Depth Rendering". Y

Fu-En Wang 83 Jan 4, 2023
Code for generating synthetic text images as described in "Synthetic Data for Text Localisation in Natural Images", Ankush Gupta, Andrea Vedaldi, Andrew Zisserman, CVPR 2016.

SynthText Code for generating synthetic text images as described in "Synthetic Data for Text Localisation in Natural Images", Ankush Gupta, Andrea Ved

Ankush Gupta 1.8k Dec 28, 2022
Code release for Hu et al., Learning to Segment Every Thing. in CVPR, 2018.

Learning to Segment Every Thing This repository contains the code for the following paper: R. Hu, P. Dollár, K. He, T. Darrell, R. Girshick, Learning

Ronghang Hu 417 Oct 3, 2022
When Age-Invariant Face Recognition Meets Face Age Synthesis: A Multi-Task Learning Framework (CVPR 2021 oral)

MTLFace This repository contains the PyTorch implementation and the dataset of the paper: When Age-Invariant Face Recognition Meets Face Age Synthesis

Hzzone 120 Jan 5, 2023
(CVPR 2021) Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds

BRNet Introduction This is a release of the code of our paper Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds,

null 86 Oct 5, 2022
Scale-aware Automatic Augmentation for Object Detection (CVPR 2021)

SA-AutoAug Scale-aware Automatic Augmentation for Object Detection Yukang Chen, Yanwei Li, Tao Kong, Lu Qi, Ruihang Chu, Lei Li, Jiaya Jia [Paper] [Bi

Jia Research Lab 182 Dec 29, 2022
(CVPR 2021) ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection

ST3D Code release for the paper ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection, CVPR 2021 Authors: Jihan Yang*, Shaoshu

CVMI Lab 224 Dec 28, 2022
Distilling Knowledge via Knowledge Review, CVPR 2021

ReviewKD Distilling Knowledge via Knowledge Review Pengguang Chen, Shu Liu, Hengshuang Zhao, Jiaya Jia This project provides an implementation for the

DV Lab 194 Dec 28, 2022
Automatically download multiple papers by keywords in CVPR

CVFPaperHelper Automatically download multiple papers by keywords in CVPR Install mkdir PapersToRead cd PaperToRead pip install requests tqdm git clon

null 46 Jun 8, 2022
Code for the paper "DewarpNet: Single-Image Document Unwarping With Stacked 3D and 2D Regression Networks" (ICCV '19)

DewarpNet This repository contains the codes for DewarpNet training. Recent Updates [May, 2020] Added evaluation images and an important note about Ma

CVLab@StonyBrook 354 Jan 1, 2023
Code for the paper STN-OCR: A single Neural Network for Text Detection and Text Recognition

STN-OCR: A single Neural Network for Text Detection and Text Recognition This repository contains the code for the paper: STN-OCR: A single Neural Net

Christian Bartz 496 Jan 5, 2023
Code for AAAI 2021 paper: Sequential End-to-end Network for Efficient Person Search

This repository hosts the source code of our paper: [AAAI 2021]Sequential End-to-end Network for Efficient Person Search. SeqNet achieves the state-of

Zj Li 218 Dec 31, 2022
Code related to "Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity" paper

DataTuner You have just found the DataTuner. This repository provides tools for fine-tuning language models for a task. See LICENSE.txt for license de

null 81 Jan 1, 2023