Implementation for Panoptic-PolarNet (CVPR 2021)

Overview

Panoptic-PolarNet

This is the official implementation of Panoptic-PolarNet.

[ArXiv paper]

Introduction

Panoptic-PolarNet is a fast and robust LiDAR point cloud panoptic segmentation framework. We learn both semantic segmentation and class-agnostic instance clustering in a single inference network using a polar Bird's Eye View (BEV) representation. Predictions from the semantic and instance head are then fused through a majority voting to create the final panopticsegmentation.

We test Panoptic-PolarNet on SemanticKITTI and nuScenes datasets. Experiment shows that Panoptic-PolarNet reaches state-of-the-art performances with a real-time inference speed.

Prepare dataset and environment

This code is tested on Ubuntu 16.04 with Python 3.8, CUDA 10.2 and Pytorch 1.7.0.

1, Install the following dependencies by either pip install -r requirements.txt or manual installation.

2, Download Velodyne point clouds and label data in SemanticKITTI dataset here.

3, Extract everything into the same folder. The folder structure inside the zip files of label data matches the folder structure of the LiDAR point cloud data.

4, Data file structure should look like this:

./
├── train.py
├── ...
└── data/
    ├──sequences
        ├── 00/           
        │   ├── velodyne/	# Unzip from KITTI Odometry Benchmark Velodyne point clouds.
        |   |	├── 000000.bin
        |   |	├── 000001.bin
        |   |	└── ...
        │   └── labels/ 	# Unzip from SemanticKITTI label data.
        |       ├── 000000.label
        |       ├── 000001.label
        |       └── ...
        ├── ...
        └── 21/
	    └── ...

5, Instance preprocessing:

python instance_preprocess.py -d </your data path> -o </preprocessed file output path>

Training

Run

python train.py

The code will automatically train, validate and save the model that has the best validation PQ.

Panoptic-PolarNet with default setting requires around 11GB GPU memory for the training. Training model on GPU with less memory would likely cause GPU out-of-memory. In this case, you can set the grid_size in the config file to [320,240,32] or lower.

Evaluate our pretrained model

We also provide a pretrained Panoptic-PolarNet weight.

python test_pretrain.py

Result will be stored in ./out folder. Test performance can be evaluated by uploading label results onto the SemanticKITTI competition website here.

Citation

Please cite our paper if this code benefits your research:

@inproceedings{Zhou2021PanopticPolarNet,
author={Zhou, Zixiang and Zhang, Yang and Foroosh, Hassan},
title={Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2021}
}

@InProceedings{Zhang_2020_CVPR,
author = {Zhang, Yang and Zhou, Zixiang and David, Philip and Yue, Xiangyu and Xi, Zerong and Gong, Boqing and Foroosh, Hassan},
title = {PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}
Comments
  • RuntimeError: Error(s) in loading state_dict for ptBEVnet:

    RuntimeError: Error(s) in loading state_dict for ptBEVnet:

    Hello, I am trying to run the code using command: python3 test_pretrain.py,

    but return: RuntimeError: Error(s) in loading state_dict for ptBEVnet: size mismatch for BEV_model.network.inc.conv.0.weight: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for BEV_model.network.inc.conv.0.bias: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for BEV_model.network.inc.conv.0.running_mean: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for BEV_model.network.inc.conv.0.running_var: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for BEV_model.network.inc.conv.1.conv1.0.weight: copying a param with shape torch.Size([64, 32, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]).

    in fact, I did not change the code of the paper.

    opened by coallar 5
  • What do labels like 131073, 65537 and 917505 mean

    What do labels like 131073, 65537 and 917505 mean

    I notice that in the prediction results there are these very big labels. I load the label file (predicted result) with

    label = np.fromfile(label_path, dtype=np.uint32, count=-1).reshape([-1,1])

    opened by l9761116 2
  • how to arrange data in data dir for test_pretrained?

    how to arrange data in data dir for test_pretrained?

    I want to do inference on some sequence. Since the space on my server is limited and I cannot even download the 80G velodyne data zip, so I copied labels and velodyne data of sequence 00 from other's server and put it in data/sequences/00. I run 'python test_pretrained.py'. But the results all showed 0.00. My velodyne dir is : /data/sequences/00/velodyne. The labels dir is: /data/sequences/00/labels. Part of the returned results are the follows: ''' 0it [00:00, ?it/s] Current val PQ is 0.000 Current val miou is 0.000 Inference time per 1 is nan seconds, postprocessing time is nan seconds per scan ''' Does anyone know if I put the data wrongly or something? Also, is it possible that I do inference on raw point cloud data?

    opened by l9761116 2
  • Check Occlusion in Data Augmentation

    Check Occlusion in Data Augmentation

    Thanks for your contribution and your work is very impressive and interesting. Just wondering about the check_occlusion function in the part of instance augmentation ( https://github.com/edwardzhou130/Panoptic-PolarNet/blob/main/dataloader/instance_augmentation.py#L160 ). The goal of this function is to check whether a point is within a (2 metre) range of any point in a point cloud right? Would dist = np.linalg.norm(points-center,axis=0) be dist = np.linalg.norm(points-center,axis=1) instead? When you compute norm on axis 0, it will be a shape of (3, ) for [x, y, z], and on axis 1 will be a shape of (N, ) for number of points.

    opened by thomas-enxuli 1
  • RuntimeError: Error(s) in loading state_dict for ptBEVnet:

    RuntimeError: Error(s) in loading state_dict for ptBEVnet:

    Hello, I am trying to run the code using command: python3 test_pretrain.py,

    but return: RuntimeError: Error(s) in loading state_dict for ptBEVnet: size mismatch for BEV_model.network.inc.conv.0.weight: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for BEV_model.network.inc.conv.0.bias: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for BEV_model.network.inc.conv.0.running_mean: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for BEV_model.network.inc.conv.0.running_var: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for BEV_model.network.inc.conv.1.conv1.0.weight: copying a param with shape torch.Size([64, 32, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]).

    in fact, I did not change the code of the paper.

    opened by coallar 0
  • Inference with Polarnet

    Inference with Polarnet

    Hi. I am trying to implement your code. I have input from Velodyne 64 as x,y,z,I. is it possible to use your model to directly use this values and obtain the predictions?

    opened by kaxapatel 1
  • Inference with custom data

    Inference with custom data

    Hi. thank you for your last response. Now I am trying to deploy your model in one of my Jetson board with Velodyne sensor. First I want to try just feeding the single frame into the model. Can you suggest where can I change for input data in the test code. so I can easily generate predictions without giving data_path. My final goal is to have real-time segmentation with Lidar input. In more simplified way, input as a lidar raw data string and output as a predictions.

    opened by kaxapatel 1
  • High loss while training on Nuscenes

    High loss while training on Nuscenes

    Hi. Really good work. I am trying to train the model but I can not debug why the losses are very high and PQ is too poor. Hope you may have some idea. Screenshot from 2022-11-02 16-06-46

    opened by kaxapatel 4
  • Not getting good results.

    Not getting good results.

    Congratulations for the great work you did. i am running the code you provided and I tested the pretrained model but why is it showing everything 0.00%. Dataloader is correct. While training it is also showing 0.00% at each iteration for validation check. Screenshot from 2022-09-23 11-28-10

    opened by kaxapatel 0
  • Error while loading nuscenes dataset

    Error while loading nuscenes dataset

    Thank you for your response @edwardzhou130. I am trying as you told me but getting this error when I try to run dataloader. I have changed lidarseg to panopticseg in line 124 Screenshot from 2022-07-24 13-05-22 .

    opened by kaxapatel 1
Owner
Zixiang Zhou
Zixiang Zhou
[arXiv'22] Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation Xiao Fu1*  Shangzhan Zhang1*  Tianrun Chen1  Yichong Lu1  Lanyun Zhu2  Xi

Xiao Fu 37 May 17, 2022
[CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)

EOPSN: Exemplar-Based Open-Set Panoptic Segmentation Network (CVPR 2021) PyTorch implementation for EOPSN. We propose open-set panoptic segmentation t

Jaedong Hwang 49 Dec 30, 2022
[CVPR 2021] Forecasting the panoptic segmentation of future video frames

Panoptic Segmentation Forecasting Colin Graber, Grace Tsai, Michael Firman, Gabriel Brostow, Alexander Schwing - CVPR 2021 [Link to paper] We propose

Niantic Labs 44 Nov 29, 2022
Pixel Consensus Voting for Panoptic Segmentation (CVPR 2020)

Implementation for Pixel Consensus Voting (CVPR 2020). This codebase contains the essential ingredients of PCV, including various spatial discretizati

Haochen 23 Oct 25, 2022
A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

Biomedical Computer Vision @ Uniandes 52 Dec 19, 2022
An unofficial personal implementation of UM-Adapt, specifically to tackle joint estimation of panoptic segmentation and depth prediction for autonomous driving datasets.

Semisupervised Multitask Learning This repository is an unofficial and slightly modified implementation of UM-Adapt[1] using PyTorch. This code primar

Abhinav Atrishi 11 Nov 25, 2022
[ICRA2021] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignment

Interactive Scene Reconstruction Project Page | Paper This repository contains the implementation of our ICRA2021 paper Reconstructing Interactive 3D

null 97 Dec 28, 2022
PanopticBEV - Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images This r

null 63 Dec 16, 2022
git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li Accepted by CVPR

NingWang 236 Dec 22, 2022
[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.

Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, CVPR 2021. Ayan Kumar Bhunia, Pinaki nath Chowdhury, Yongxin Yan

Ayan Kumar Bhunia 44 Dec 12, 2022
[CVPR 2022] CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation

CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation Prerequisite Please create and activate the following conda envrionment. To r

Qin Wang 87 Jan 8, 2023
PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Neural Scene Flow Fields PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 20

Zhengqi Li 585 Jan 4, 2023
Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

IC-Conv This repository is an official implementation of the paper Inception Convolution with Efficient Dilation Search. Getting Started Download Imag

Jie Liu 111 Dec 31, 2022
Official implementation for (Refine Myself by Teaching Myself : Feature Refinement via Self-Knowledge Distillation, CVPR-2021)

FRSKD Official implementation for Refine Myself by Teaching Myself : Feature Refinement via Self-Knowledge Distillation (CVPR-2021) Requirements Pytho

null 75 Dec 28, 2022
Official PyTorch implementation of RobustNet (CVPR 2021 Oral)

RobustNet (CVPR 2021 Oral): Official Project Webpage Codes and pretrained models will be released soon. This repository provides the official PyTorch

Sungha Choi 173 Dec 21, 2022
PyTorch implementation for COMPLETER: Incomplete Multi-view Clustering via Contrastive Prediction (CVPR 2021)

Completer: Incomplete Multi-view Clustering via Contrastive Prediction This repo contains the code and data of the following paper accepted by CVPR 20

XLearning Group 72 Dec 7, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
Implementation for the paper SMPLicit: Topology-aware Generative Model for Clothed People (CVPR 2021)

SMPLicit: Topology-aware Generative Model for Clothed People [Project] [arXiv] License Software Copyright License for non-commercial scientific resear

Enric Corona 225 Dec 13, 2022
The official implementation of our CVPR 2021 paper - Hybrid Rotation Averaging: A Fast and Robust Rotation Averaging Approach

Graph Optimizer This repo contains the official implementation of our CVPR 2021 paper - Hybrid Rotation Averaging: A Fast and Robust Rotation Averagin

Chenyu 109 Dec 23, 2022