Semi-supervised Implicit Scene Completion from Sparse LiDAR
Created by Pengfei Li, Yongliang Shi, Tianyu Liu, Hao Zhao, Guyue Zhou and YA-QIN ZHANG from Institute for AI Industry Research(AIR), Tsinghua University.
For complete video, click HERE.
Introduction
Recent advances show that semi-supervised implicit representation learning can be achieved through physical constraints like Eikonal equations. However, this scheme has not yet been successfully used for LiDAR point cloud data, due to its spatially varying sparsity.
In this repository, we develop a novel formulation that conditions the semi-supervised implicit function on localized shape embeddings. It exploits the strong representation learning power of sparse convolutional networks to generate shape-aware dense feature volumes, while still allows semi-supervised signed distance function learning without knowing its exact values at free space. With extensive quantitative and qualitative results, we demonstrate intrinsic properties of this new learning system and its usefulness in real-world road scenes. Notably, we improve IoU from 26.3% to 51.0% on SemanticKITTI. Moreover, we explore two paradigms to integrate semantic label predictions, achieving implicit semantic completion. Codes and data are publicly available.
Citation
If you find our work useful in your research, please consider citing:
###to do###
Installation
Requirements
CUDA=11.1
python>=3.8
Pytorch>=1.8
numpy
ninja
MinkowskiEngine
tensorboard
pyyaml
configargparse
scripy
open3d
h5py
plyfile
scikit-image
Clone the repository:
git clone https://github.com/OPEN-AIR-SUN/SISC.git
Data preparation
Download the SemanticKITTI dataset from HERE. Unzip it into the same directory as SISC.
Training and inference
The configuration for training/inference is stored in opt.yaml, which can be modified as needed.
Scene Completion
Run the following command for a certain task (train/valid/visualize):
CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 main_sc.py --task=[task] --experiment_name=[experiment_name]
Semantic Scene Completion
SSC option A
Run the following command for a certain task (ssc_pretrain/ssc_valid/train/valid/visualize):
CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 main_ssc_a.py --task=[task] --experiment_name=[experiment_name]
Here, use ssc_pretrain/ssc_valid to train/validate the SSC part. Then the pre-trained model can be used to further train the whole model.
SSC option B
Run the following command for a certain task (train/valid/visualize):
CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 main_ssc_b.py --task=[task] --experiment_name=[experiment_name]
Model Zoo
Our pre-trained models can be downloaded here:
| Ablation | Pretrained Checkpoints | |||||
| data augmentation | no aug | rotate & flip | ||||
| Dnet input | radial distance | radial distance & height | ||||
| Dnet structure | last1 pruning | last2 pruning | last3 pruning | last4 pruning | Dnet relu | 4convs output |
| Gnet structure | width128 depth4 | width512 depth4 | width256 depth3 | width256 depth5 | Gnet relu | |
| point sample | on:off=1:2 | on:off=2:3 | ||||
| positional encoding | no encoding | incF level10 | incT level5 | incT level15 | ||
| sample strategy | nearest | |||||
| scale size | scale 2 | scale 4 | scale 8 | scale 16 | scale 32 | |
| shape size | shape 128 | shape 512 | ||||
| SSC | SSC option A | SSC option B | ||||
These models correspond to the ablation study in our paper. The Scale 4 works as our baseline.
43 Dec 12, 2022
19 Oct 22, 2022
35 Nov 20, 2022
302 Dec 14, 2022
51 Nov 29, 2022
49 Jan 2, 2023
2 Dec 17, 2021
5 Dec 10, 2022
149 Jan 8, 2023
243 Jan 7, 2023
6 Oct 28, 2022
181 Dec 29, 2022
102 Dec 25, 2022
10 Oct 11, 2022
224 Dec 28, 2022
182 Dec 30, 2022
33 Dec 12, 2022
151 Dec 26, 2022