Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather

Related tags

Deep Learning computer-vision simulation fog lidar lidar-point-cloud fog-removal fog-attenuation
Overview

LiDAR fog simulation

PWC

Created by Martin Hahner at the Computer Vision Lab of ETH Zurich.

This is the official code release of the paper
Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather
by Martin Hahner, Christos Sakaridis, Dengxin Dai, and Luc van Gool, accepted at ICCV 2021.

Please visit our paper website for more details.

pointcloud_viewer

Overview

.
├── file_lists                          # contains file lists for pointcloud_viewer.py
│   └── ...
├── integral_lookup_tables              # contains lookup tables to speed up the fog simulation
│   └── ... 
├── extract_fog.py                      # to extract real fog noise* from the SeeingThroughFog dataset
├── fog_simulation.py                   # to augment a clear weather pointcloud with artificial fog (used during training)
├── generate_integral_lookup_table.py   # to precompute the integral inside the fog equation
├── pointcloud_viewer.py                # to visualize entire point clouds of different datasets with the option to augment fog into their scenes
├── README.md
└── theory.py                           # to visualize the theory behind a single LiDAR beam in foggy conditions

* Contains returns not only from fog, but also from physical objects that are closeby.

Datasets supported by pointcloud_viewer.py:

License

This software is made available for non-commercial use under a Creative Commons License.
A summary of the license can be found here.

Acknowledgments

This work is supported by Toyota via the TRACE project.

Furthermore, we would like to thank the authors of SeeingThroughFog for their great work.
In this repository, we use a fork of their original repository to visualize annotations and compare to their fog simulation. Their code is licensed via the MIT License.

Citation

If you find this work useful, please consider citing our paper.

@inproceedings{HahnerICCV21,
  author = {Hahner, Martin and Sakaridis, Christos and Dai, Dengxin and Van Gool, Luc},
  title = {Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather},
  booktitle = {IEEE International Conference on Computer Vision (ICCV)},
  year = {2021},
}

Getting Started

Setup

  1. Install anaconda.

  2. Create a new conda environment.

conda create --name foggy_lidar python=3.9 -y
  1. Activate the newly created conda environment.
conda activate foggy_lidar
  1. Install all necessary packages.
conda install matplotlib numpy opencv pandas plyfile pyopengl pyqt pyqtgraph quaternion scipy tqdm -c conda-forge -y
pip install pyquaternion
  1. Clone this repository (including submodules).
git clone [email protected]:MartinHahner/LiDAR_fog_sim.git --recursive
cd LiDAR_fog_sim

Usage

How to run the script that visualizes the theory behind a single LiDAR beam in foggy conditions:

python theory.py

theory

How to run the script that visualizes entire point clouds of different datasets:

python pointcloud_viewer.py -d <path_to_where_you_store_your_datasets>

Note:

You may also have to adjust the relative paths in pointcloud_viewer.py (right at the beginning of the file) to be compatible with your datasets relative folder structure.

Disclaimer

The code has been successfully tested on

  • Ubuntu 18.04.5 LTS
  • macOS Big Sur 11.2.1
  • Debian GNU/Linux 9.13

using conda 4.9.2.

Contributions

Please feel free to suggest improvements to this repository.
We are always open to merge usefull pull request.

Comments
  • the wrong results of fog simulation on waymo and Kitti

    the wrong results of fog simulation on waymo and Kitti

    Thank you for your excellent work.

    In fact, it is similar to issue "the wrong result of the fog\u simulation.py using the Kitti data. \7". I directly run the "fog_simulation.py" file on Kitti and waymo datasets. The results are similar to issue 7. It seems that almost all points are regarded as fog noise.

    I want to know whether it needs to be modified due to the intensity of the data or whether it is impossible to directly run fog_ simulation. Instead, I need to run pointcloud_viewer.py (doesn't sound reasonable)

    Looking forward to your reply

    stale 
    opened by ylwhxht 2
  • The wrong result of the fog_simulation.py using the kitti data.

    The wrong result of the fog_simulation.py using the kitti data.

    Thank you for your excellent contribution of this work！There is a problem when I run the fog_simulation.py to generate the fog by using the kitti dataset. 20220505-092622 20220505-091752 The 1st picture is the original kitti data. The 2nd picture is generated by the fog_simulation.py. I wonder how can i solve the problem,thank you.

    opened by zzqjh 2
  • Problem about loading openpcdet result.pkl

    Problem about loading openpcdet result.pkl

    Hi! I came from OpenPCDet issues channel and found your amazing work! But I have a problem about loading the result.pkl with kitti velodyne data. I can't visualize the predicted bbox in the point cloud. Is there anything wrong with my file path with the pickle? Thanks a lot in advance! The interface is really great. image

    opened by zitgit 2
  • Visualizing output of demo.py of OpenPCDet using a Headless Server

    Visualizing output of demo.py of OpenPCDet using a Headless Server

    Hello First of all, thank you for these codes, they've helped me a lot. I am a newbie to 3D object detection, I saw the demo.py code that you wrote in OpenPCDet, I have used it and got as an output the data_dict_idx.pkl and pred_dicts_idx.pkl . Now I am trying to visualize these outputs using your pointcloud_viewer but when i choose the custom directory where i put the demo.py output, nothing shows up, I was wondering if you could help me

    opened by udacityyy 1
  • Does the simulation results match real world data?

    Does the simulation results match real world data?

    Thanks for your excellent work. I am hoping to use the simulation to generate some synthetic foggy data, e.g. on KITTI. Hence, I am wondering how much does the simulated data match with real-world foggy data? But I found only some minor qualitive discussion is represented in the paper, no quantitive discussion.

    question 
    opened by tdzdog 1
  • Question about evaluation on dense fog weather in STF Dataset

    Question about evaluation on dense fog weather in STF Dataset

    Hi Martin,

    Thank you for providing this elegant and powerful repo and I really appreciate your brilliant work!

    I'm trying to reproduce the training pipeline on Seeing Through Fog Dataset. In the experimental part of the paper, there are clear weather baseline and "strongest ∩ last filter" baseline. I want to know whether clear weather baseline is evaluated on strongest return or last return of dense fog test split. I don't seem to find a description in the paper or in the supplementary materials, so sorry if I missed it.

    Thank you in advance!

    opened by qifang-robotics 1
Owner
Martin Hahner
PhD Candidate
Martin Hahner
GitHub https://www.trace.ethz.ch/lidar_fog_sim
Image-Adaptive YOLO for Object Detection in Adverse Weather Conditions

Image-Adaptive YOLO for Object Detection in Adverse Weather Conditions Accepted by AAAI 2022 [arxiv] Wenyu Liu, Gaofeng Ren, Runsheng Yu, Shi Guo, Jia

liuwenyu 245 Dec 16, 2022
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (CVPR 2022, Oral)

Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (CVPR 2022, Oral) This is the official implementat

Yifan Zhang 259 Dec 25, 2022
A general python framework for single object tracking in LiDAR point clouds, based on PyTorch Lightning.

Open3DSOT A general python framework for single object tracking in LiDAR point clouds, based on PyTorch Lightning. The official code release of BAT an

Kangel Zenn 172 Dec 23, 2022
Code of Adverse Weather Image Translation with Asymmetric and Uncertainty aware GAN

Adverse Weather Image Translation with Asymmetric and Uncertainty-aware GAN (AU-GAN) Official Tensorflow implementation of Adverse Weather Image Trans

Jeong-gi Kwak 36 Dec 26, 2022
Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving

SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving Abstract In this paper, we introduce SalsaNext f

null 308 Jan 4, 2023
(CVPR 2021) Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds

BRNet Introduction This is a release of the code of our paper Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds,

null 86 Oct 5, 2022
Code for the paper SphereRPN: Learning Spheres for High-Quality Region Proposals on 3D Point Clouds Object Detection, ICIP 2021.

SphereRPN Code for the paper SphereRPN: Learning Spheres for High-Quality Region Proposals on 3D Point Clouds Object Detection, ICIP 2021. Authors: Th

Thang Vu 15 Dec 2, 2022
BADet: Boundary-Aware 3D Object Detection from Point Clouds (Pattern Recognition 2022)

BADet: Boundary-Aware 3D Object Detection from Point Clouds (Pattern Recognition

Rui Qian 17 Dec 12, 2022
Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds (CVPR 2022)

Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds (CVPR2022)[paper] Authors: Chenhang He, Ruihuang Li, Shuai Li, L

Billy HE 141 Dec 30, 2022
Code for "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds", CVPR 2021

PV-RAFT This repository contains the PyTorch implementation for paper "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clou

Yi Wei 43 Dec 5, 2022
Synthetic LiDAR sequential point cloud dataset with point-wise annotations

SynLiDAR dataset: Learning From Synthetic LiDAR Sequential Point Cloud This is official repository of the SynLiDAR dataset. For technical details, ple

null 78 Dec 27, 2022
This is a package for LiDARTag, described in paper: LiDARTag: A Real-Time Fiducial Tag System for Point Clouds

LiDARTag Overview This is a package for LiDARTag, described in paper: LiDARTag: A Real-Time Fiducial Tag System for Point Clouds (PDF)(arXiv). This wo

University of Michigan Dynamic Legged Locomotion Robotics Lab 159 Dec 21, 2022
CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection

CLOCs is a novel Camera-LiDAR Object Candidates fusion network. It provides a low-complexity multi-modal fusion framework that improves the performance of single-modality detectors. CLOCs operates on the combined output candidates of any 3D and any 2D detector, and is trained to produce more accurate 3D and 2D detection results.

Su Pang 254 Dec 16, 2022
Code for "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" @ICRA2021

CloudAAE This is an tensorflow implementation of "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" Files log:

Gee 35 Nov 14, 2022
The official implementation of ICCV paper "Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds".

Box-Aware Tracker (BAT) Pytorch-Lightning implementation of the Box-Aware Tracker. Box-Aware Feature Enhancement for Single Object Tracking on Point C

Kangel Zenn 5 Mar 26, 2022
This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection, built on SECOND.

3D-CVF This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object

YecheolKim 97 Dec 20, 2022
OpenPCDet Toolbox for LiDAR-based 3D Object Detection.

OpenPCDet OpenPCDet is a clear, simple, self-contained open source project for LiDAR-based 3D object detection. It is also the official code release o

OpenMMLab 3.2k Dec 31, 2022
LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection

LiDAR Distillation Paper | Model LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection Yi Wei, Zibu Wei, Yongming Rao, Jiax

Yi Wei 75 Dec 22, 2022
This project is the official implementation of our accepted ICLR 2021 paper BiPointNet: Binary Neural Network for Point Clouds.

BiPointNet: Binary Neural Network for Point Clouds Created by Haotong Qin, Zhongang Cai, Mingyuan Zhang, Yifu Ding, Haiyu Zhao, Shuai Yi, Xianglong Li

Haotong Qin 59 Dec 17, 2022