SNE-RoadSeg in PyTorch, ECCV 2020

Overview

SNE-RoadSeg

Introduction

This is the official PyTorch implementation of SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection, accepted by ECCV 2020. This is our project page.

In this repo, we provide the training and testing setup for the KITTI Road Dataset. We test our code in Python 3.7, CUDA 10.0, cuDNN 7 and PyTorch 1.1. We provide Dockerfile to build the docker image we use.

Setup

Please setup the KITTI Road Dataset and pretrained weights according to the following folder structure:

SNE-RoadSeg
 |-- checkpoints
 |  |-- kitti
 |  |  |-- kitti_net_RoadSeg.pth
 |-- data
 |-- datasets
 |  |-- kitti
 |  |  |-- training
 |  |  |  |-- calib
 |  |  |  |-- depth_u16
 |  |  |  |-- gt_image_2
 |  |  |  |-- image_2
 |  |  |-- validation
 |  |  |  |-- calib
 |  |  |  |-- depth_u16
 |  |  |  |-- gt_image_2
 |  |  |  |-- image_2
 |  |  |-- testing
 |  |  |  |-- calib
 |  |  |  |-- depth_u16
 |  |  |  |-- image_2
 |-- examples
 ...

image_2, gt_image_2 and calib can be downloaded from the KITTI Road Dataset. We implement depth_u16 based on the LiDAR data provided in the KITTI Road Dataset, and it can be downloaded from here. Note that depth_u16 has the uint16 data format, and the real depth in meters can be obtained by double(depth_u16)/1000. Moreover, the pretrained weights kitti_net_RoadSeg.pth for our SNE-RoadSeg-152 can be downloaded from here.

Usage

Run an example

We provide one example in examples. To run it, you only need to setup the checkpoints folder as mentioned above. Then, run the following script:

bash ./scripts/run_example.sh

and you will see normal.png, pred.png and prob_map.png in examples. normal.png is the normal estimation by our SNE; pred.png is the freespace prediction by our SNE-RoadSeg; and prob_map.png is the probability map predicted by our SNE-RoadSeg.

Testing for KITTI submission

For KITTI submission, you need to setup the checkpoints and the datasets/kitti/testing folder as mentioned above. Then, run the following script:

bash ./scripts/test.sh

and you will get the prediction results in testresults. After that you can follow the submission instructions to transform the prediction results into the BEV perspective for submission.

If everything works fine, you will get a MaxF score of 96.74 for URBAN. Note that this is our re-implemented weights, and it is very similar to the reported ones in the paper (a MaxF score of 96.75 for URBAN).

Training on the KITTI dataset

For training, you need to setup the datasets/kitti folder as mentioned above. You can split the original training set into a new training set and a validation set as you like. Then, run the following script:

bash ./scripts/train.sh

and the weights will be saved in checkpoints and the tensorboard record containing the loss curves as well as the performance on the validation set will be save in runs. Note that use-sne in train.sh controls if we will use our SNE model, and the default is True. If you delete it, our RoadSeg will take depth images as input, and you also need to delete use-sne in test.sh to avoid errors when testing.

Citation

If you use this code for your research, please cite our paper.

@inproceedings{fan2020sne,
  author = {Fan, Rui and Wang, Hengli and Cai, Peide and Liu, Ming},
  title = {SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year = {2020},
  organization = {Springer},
}

Acknowledgement

Our code is inspired by pytorch-CycleGAN-and-pix2pix, and we thank Jun-Yan Zhu for their great work.

Comments
  • How to generate dense depth images from lidar point cloud?

    How to generate dense depth images from lidar point cloud?

    Hello Wang: @hlwang1124 I found that the generated dense depth image may influence the inference performance. Can you provide the dense depth image generation code for us? I try your method mentioned in #7. But it seems that generated result is different from your privided depth_u16 data. My EMAIL: [email protected]. Thanks a lot!

    opened by mei123hao 27
  • The problem of using self-built dataset to train models and test self-built dataset

    The problem of using self-built dataset to train models and test self-built dataset

    Hi, I am a newbie in semantic segmentation. Can the images marked with labelme be used as a dataset? If so, how to create a dataset and how to use it, do I need to retrain the model with a self-built dataset? How can the trained model be used for testing? I wait for your response.

    opened by mjw123bs 10
  • Question about the quality of the result

    Question about the quality of the result

    Thank you for your excellent work!

    I got the result as this https://github.com/hlwang1124/SNE-RoadSeg/issues/17#issuecomment-733799715. The following is disparity, normal, prob_map image image image

    As you can see, the final result is worse than your one in https://github.com/hlwang1124/SNE-RoadSeg/issues/17#issuecomment-733799715.

    Is the quality of my results normal?

    Could you give me some advice how to improve the quality?

    Thank you in advance!

    opened by linpeisensh 8
  • Question on the test results on my own data and the training dataset choices

    Question on the test results on my own data and the training dataset choices

    Hello, I tried to use the data captured by ZED 2 depth camera ,which are shown below, to obtain the road prediction. But the network doesn't work at all. Do you have any ideas on the results? And I noticed that in your paper, three datasets are used to train the model, and the code you show here is only for KITTI dataset? Thank you in advance! 1 normal pred prob_map rgb

    opened by SimonSongg 8
  • R2D dataset issue

    R2D dataset issue

    Hi,

    I tried to download the R2D dataset and to train. But the labels of this dataset seem to be all in black. And the depth images have many strange texture. Should it be like this? Thank you very much. 000000 image

    opened by SimonSongg 8
  • The problem of the R2D dataset

    The problem of the R2D dataset

    Hello! I downloaded the R2D dataset you published before, and I want to use it further. Where can I get more information about this dataset? For example, what category does the label of the dataset correspond to, some corresponding settings, and so on.
    Thank you very much!!!

    opened by lrjing 7
  • Test result is nan

    Test result is nan

    Thanks again for your previous reply!

    After the env been set up(datasets, pretrained model), I run the test script: bash ./scripts/test.sh the F_score is nan as shown below:

    model [RoadSegModel] was created
    loading the model from ./checkpoints/kitti/kitti_net_RoadSeg.pth
    ---------- Networks initialized -------------
    [Network RoadSeg] Total number of parameters : 201.325 M
    -----------------------------------------------
    Epoch kitti test loss: 0.838 loss: 0.569 
    Epoch kitti glob acc : 0.821, pre : 0.000, recall : nan, F_score : nan, IoU : 0.000
    

    No error occurred, Would you please enlighten me on which part probably went wrong?

    Thanks in advance!

    opened by zeng32 7
  • Questions regarding the evaluation

    Questions regarding the evaluation

    1. Would you please share your evaluation code(devkit_road) with us? the original code downloaded from the http://www.cvlibs.net/datasets/kitti/eval_road.php uses very old dependencies: python 2.7/numpy 1.51/opencv 2.4.6. It is almost impossible to rebuild this kind of environment. Do you also use the original code? if not, would you please share your new code with us?

    2. The evaluation method which you are using in the training process is different from the evaluation code in the devkit_road, right? Therefore, the evaluation result showing in the training process can only be regarded as an indicator and can not be used as the result for submission to the server, am I correct?

    Thanks in advance!

    opened by zeng32 6
  • The predict image is black without any data?

    The predict image is black without any data?

    Hello! I try to run the example scripts. However, the predicted image is black without any data. I try to run the test scripts but I get the nan results? What wrong about that ? Anyone tell me Please ? Thanks!!!

    opened by mei123hao 5
  • How can we generate the dense depth image from sparse LiDAR points?

    How can we generate the dense depth image from sparse LiDAR points?

    Hi Hengli, Could you provide some instructions for generating dense depth maps? Using depth completion networks or simply using interpolation? If we don't have perfect depth maps, could SNE and RoadSeg Net generalize well?

    opened by ZhenboSong 5
  • A problem regarding testing the model on kitti dataset

    A problem regarding testing the model on kitti dataset

    Hello, I try to run "run_example.py" to test it on the image in the test dataset and it works very well. Great work!! But when I tried to run "test.py", an error appeared.

    ---------- Networks initialized ------------- [Network RoadSeg] Total number of parameters : 201.325 M

    Traceback (most recent call last): File "test.py", line 31, in for i, data in enumerate(dataset): File "C:\Users\sf995\iCloudDrive\DL\SNE-RoadSeg\data_init_.py", line 71, in iter for i, data in enumerate(self.dataloader): File "D:\anaconda\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 193, in iter return _DataLoaderIter(self) File "D:\anaconda\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 469, in init w.start() File "D:\anaconda\envs\pytorch\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "D:\anaconda\envs\pytorch\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "D:\anaconda\envs\pytorch\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "D:\anaconda\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child) File "D:\anaconda\envs\pytorch\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'CustomDatasetDataLoader.initialize..'

    (pytorch) C:\Users\sf995\iCloudDrive\DL\SNE-RoadSeg>Traceback (most recent call last): File "", line 1, in File "D:\anaconda\envs\pytorch\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "D:\anaconda\envs\pytorch\lib\multiprocessing\spawn.py", line 115, in _main self = reduction.pickle.load(from_parent) EOFError: Ran out of input

    Seems like it is about the lambda function. Could you help me?

    opened by SimonSongg 4
  • SNE-RoadSeg+ issue

    SNE-RoadSeg+ issue

    Hello. I would like to utilize the result of SNE-RoadSeg+ (IROS 2021). However, I could not reproduce your result. Would you give me trained parameter file or code of SNE-RoadSeg+?

    opened by MaryAhn 0
  • Questions about sne

    Questions about sne

    What I understand is that when sne finds the optimal normal direction, it ignores the fluctuations in the xy direction and only optimizes the z direction. Is this understanding correct, and if so, why this is possible

    opened by xmqv0913 0
  • Questions on SNE-RoadSeg+

    Questions on SNE-RoadSeg+

    Hi, after reading your paper (https://arxiv.org/pdf/2107.14599.pdf) I have a question on SNE-RoadSeg+ you proposed. The solution to the extreme problem in equation (3) is not so trivial. Can you give some derivations? Thanks so much.

    opened by Jiayi719 0
  • usage of depth image in R2D dataset

    usage of depth image in R2D dataset

    In the R2D dataset, I read the depth image as an RGB image, and followed the instructions in README to convert the image with 3 channel into depth value, and stored it using uint16. But the result seems incorrect, because the depth value is not continuous on the road. Could you tell me how to use the depth image correctly? Thanks a lot. origin_image converted_depth .

    opened by ybWang0820 0
Owner
Ph.D. candidate in HKUST, supervised by Prof.Ming Liu, a member of RAM-LAB, Robotics Institute
null
PyTorch implementation of ECCV 2020 paper "Foley Music: Learning to Generate Music from Videos "

Foley Music: Learning to Generate Music from Videos This repo holds the code for the framework presented on ECCV 2020. Foley Music: Learning to Genera

Chuang Gan 30 Nov 3, 2022
Unofficial PyTorch implementation of "RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving" (ECCV 2020)

RTM3D-PyTorch The PyTorch Implementation of the paper: RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving (ECCV 2020

Nguyen Mau Dzung 271 Nov 29, 2022
The implementation for paper Joint t-SNE for Comparable Projections of Multiple High-Dimensional Datasets.

Joint t-sne This is the implementation for paper Joint t-SNE for Comparable Projections of Multiple High-Dimensional Datasets. abstract: We present Jo

IDEAS Lab 7 Dec 18, 2022
Company clustering with K-means/GMM and visualization with PCA, t-SNE, using SSAN relation extraction

RE results graph visualization and company clustering Installation pip install -r requirements.txt python -m nltk.downloader stopwords python3.7 main.

Jieun Han 1 Oct 6, 2022
[ECCV 2020] Reimplementation of 3DDFAv2, including face mesh, head pose, landmarks, and more.

Stable Head Pose Estimation and Landmark Regression via 3D Dense Face Reconstruction Reimplementation of (ECCV 2020) Towards Fast, Accurate and Stable

Remilia Scarlet 221 Dec 30, 2022
1st Place Solution to ECCV-TAO-2020: Detect and Represent Any Object for Tracking

Instead, two models for appearance modeling are included, together with the open-source BAGS model and the full set of code for inference. With this code, you can achieve around mAP@23 with TAO test set (based on our estimation).

null 79 Oct 8, 2022
Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Causality In Traffic Accident (Under Construction) Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020) Overview Data Prepa

Tackgeun 21 Nov 20, 2022
git《Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction》(ECCV 2020) GitHub:

Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction Code for the ECCV 2020 paper by Yiming Qian and Yasutaka Furukawa Getting

null 37 Dec 4, 2022
Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

PWLQ Updates 2020/07/16 - We are working on getting permission from our institution to release our source code. We will release it once we are granted

null 54 Dec 15, 2022
dataset for ECCV 2020 "Motion Capture from Internet Videos"

Motion Capture from Internet Videos Motion Capture from Internet Videos Junting Dong*, Qing Shuai*, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, Hujun Bao

ZJU3DV 98 Dec 7, 2022
Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

Adversarial Training Against Location-Optimized Adversarial Patches arXiv | Paper | Code | Video | Slides Code for the paper: Sukrut Rao, David Stutz,

Sukrut Rao 32 Dec 13, 2022
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

null 42 Nov 24, 2022
Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh

Arjun Majumdar 44 Dec 14, 2022
Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Contact and Human Dynamics from Monocular Video This is the official implementation for the ECCV 2020 spotlight paper by Davis Rempe, Leonidas J. Guib

Davis Rempe 207 Jan 5, 2023
[ECCV 2020] Gradient-Induced Co-Saliency Detection

Gradient-Induced Co-Saliency Detection Zhao Zhang*, Wenda Jin*, Jun Xu, Ming-Ming Cheng ⭐ Project Home » The official repo of the ECCV 2020 paper Grad

Zhao Zhang 35 Nov 25, 2022
Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

Martin Li 85 Dec 22, 2022
Code for paper ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop.

Who Left the Dogs Out? Evaluation and demo code for our ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization

Benjamin Biggs 29 Dec 28, 2022
Sign Language Translation with Transformers (COLING'2020, ECCV'20 SLRTP Workshop)

transformer-slt This repository gathers data and code supporting the experiments in the paper Better Sign Language Translation with STMC-Transformer.

Kayo Yin 107 Dec 27, 2022
Source code for "Progressive Transformers for End-to-End Sign Language Production" (ECCV 2020)

Progressive Transformers for End-to-End Sign Language Production Source code for "Progressive Transformers for End-to-End Sign Language Production" (B

null 58 Dec 21, 2022