Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

Related tags

Deep Learning Point-Spatio-Temporal-Convolution
Overview

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences

Introduction

Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In the paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences. Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension. Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of 3D point cloud sequences in a spatio-temporally hierarchical manner.

Installation

The code is tested with Red Hat Enterprise Linux Workstation release 7.7 (Maipo), g++ (GCC) 8.3.1, PyTorch v1.2, CUDA 10.2 and cuDNN v7.6.

Install PyTorch v1.2:

pip install torch==1.2.0 torchvision==0.4.0

Compile the CUDA layers for PointNet++, which we used for furthest point sampling (FPS) and radius neighbouring search:

cd modules
python setup.py install

To see if the compilation is successful, try to run python modules/pst_convolutions.py to see if a forward pass works.

Install Mayavi for point cloud visualization (optional). Desktop is required.

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{fan2021pstnet,
    title={PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences},
    author={Hehe Fan and Xin Yu and Yuhang Ding and Yi Yang and Mohan Kankanhalli},
    booktitle={International Conference on Learning Representations},
    year={2021}
}

Related Repos

  1. PointNet++ PyTorch implementation: https://github.com/facebookresearch/votenet/tree/master/pointnet2
  2. MeteorNet: https://github.com/xingyul/meteornet
  3. 3DV: https://github.com/3huo/3DV-Action
Comments
  • Default result only got 84 on MSRA

    Default result only got 84 on MSRA

    Great works! I've read it thoroughly and can easily run your code.

    One issue is when I directly use the train-msr and finish train, the final acc is 84 and 83 for two repeat. I think they are with 16 frames. My repeats get lower scoring comparing to the paper. Screen Shot 2021-06-08 at 9 27 35 AM Screen Shot 2021-06-08 at 9 27 29 AM

    I think it might because there are some parameter change in the script. Would you be convenient to check out?

    opened by Jarrome 12
  • PSTNET on SemanticKitti/nuScenes dataset

    PSTNET on SemanticKitti/nuScenes dataset

    Great work on the Point Tubes. I am particularly interested in the 4D semantic segmentation applications. I was wondering if you tried the PSTNet on benchmark datasets like Semantic Kitti or nuScenes dataset. These pointcloud sequences are much more sparse than the SYNTHIA dataset.

    Thank you

    opened by sandeepnmenon 6
  • An Issue related to running the furthest_point_sampling function

    An Issue related to running the furthest_point_sampling function

    Dear Author,

    Hope you are well.

    I got an issue related to the pointnet2_utils.py and setup.py files. When I stalled setup.py the file, I got a warning below related to the GCC version.

    b2f965312567d709789efabf3f0db5d

    Then when I was running the highlighted part within the setup.py below, I got an error of "Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)".

    image

    Just wondering if you happen to know what's the issue? Does it related to the g++version?

    Our PyTorch version is 1.2.0.

    Many thanks! @hehefan

    opened by NeilCui6 4
  • Training on NTU 60 XSUB

    Training on NTU 60 XSUB

    Thanks for your awesome work and generous code sharing. Recently, I have tried to reproduce the results on NTU 60 XSUB benchmark with one 3090 GPU. However, it seems the training is really slow and it would take more than a week to train 20 epochs following the original setting. I have tried to enlarge the batch size, and it still quite slow. So, what is the reasonable training time for NTU 60 dataset? Is there anything I have missed?

    Looking forward to your reply.

    Thanks

    opened by erinchen824 3
  • Request for the MSR data set in npy format

    Request for the MSR data set in npy format

    Hi Hehe,

    Can you pleaase share the npy data file for the MSR data set. It is a bit struggle for me to phrase bin data into npy. Hope you can help me.

    Best, Xin

    opened by Delusionx1 1
  • about usage

    about usage

    Hi, @hehefan ,

    Thanks for releasing the package. Would you also release a basic step-by-step guide for the usage of PSTConv? e.g, data preparation, training steps and prediction steps...

    Thanks~

    opened by amiltonwong 1
Owner
Hehe Fan
Research fellow at the National University of Singapore.
Hehe Fan
GitHub
Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

NeuralTextures This is repository with inference code for paper "StylePeople: A Generative Model of Fullbody Human Avatars" (CVPR21). This code is for

Visual Understanding Lab @ Samsung AI Center Moscow 18 Oct 6, 2022
Code for paper ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop.

Who Left the Dogs Out? Evaluation and demo code for our ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization

Benjamin Biggs 29 Dec 28, 2022
The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

Leo Xiao 3.9k Jan 5, 2023
Home repository for the Regularized Greedy Forest (RGF) library. It includes original implementation from the paper and multithreaded one written in C++, along with various language-specific wrappers.

Regularized Greedy Forest Regularized Greedy Forest (RGF) is a tree ensemble machine learning method described in this paper. RGF can deliver better r

RGF-team 364 Dec 28, 2022
Official implementation of AAAI-21 paper "Label Confusion Learning to Enhance Text Classification Models"

Description: This is the official implementation of our AAAI-21 accepted paper Label Confusion Learning to Enhance Text Classification Models. The str

null 101 Nov 25, 2022
Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

Context Matters: Graph-based Self-supervised Representation Learning for Medical Images Official PyTorch implementation for paper Context Matters: Gra

null 49 Nov 23, 2022
A PyTorch re-implementation of the paper 'Exploring Simple Siamese Representation Learning'. Reproduced the 67.8% Top1 Acc on ImageNet.

Exploring simple siamese representation learning This is a PyTorch re-implementation of the SimSiam paper on ImageNet dataset. The results match that

Taojiannan Yang 72 Nov 9, 2022
Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting.

Non-AR Spatial-Temporal Transformer Introduction Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series For

Chen Kai 66 Nov 28, 2022
This is a Pytorch implementation of the paper: Self-Supervised Graph Transformer on Large-Scale Molecular Data.

This is a Pytorch implementation of the paper: Self-Supervised Graph Transformer on Large-Scale Molecular Data.

null 212 Dec 25, 2022
Official implementation of the ICLR 2021 paper

You Only Need Adversarial Supervision for Semantic Image Synthesis Official PyTorch implementation of the ICLR 2021 paper "You Only Need Adversarial S

Bosch Research 272 Dec 28, 2022
Implementation of Nyström Self-attention, from the paper Nyströmformer

Nyström Attention Implementation of Nyström Self-attention, from the paper Nyströmformer. Yannic Kilcher video Install $ pip install nystrom-attention

Phil Wang 95 Jan 2, 2023
Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

SETR - Pytorch Since the original paper (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.) has no official

zhaohu xing 112 Dec 16, 2022
Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

CIPS -- Official Pytorch Implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis Requirements pip install -r requi

Multimodal Lab @ Samsung AI Center Moscow 201 Dec 21, 2022
Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

HiSD: Image-to-image Translation via Hierarchical Style Disentanglement Official pytorch implementation of paper "Image-to-image Translation

null 364 Dec 14, 2022
PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Neural Scene Flow Fields PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 20

Zhengqi Li 585 Jan 4, 2023
Implementation of Barlow Twins paper

barlowtwins PyTorch Implementation of Barlow Twins paper: Barlow Twins: Self-Supervised Learning via Redundancy Reduction This is currently a work in

IgorSusmelj 86 Dec 20, 2022
Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

IC-Conv This repository is an official implementation of the paper Inception Convolution with Efficient Dilation Search. Getting Started Download Imag

Jie Liu 111 Dec 31, 2022
Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

LLA: Loss-aware Label Assignment for Dense Pedestrian Detection This project provides an implementation for "LLA: Loss-aware Label Assignment for Dens

null 35 Dec 6, 2022
Functional TensorFlow Implementation of Singular Value Decomposition for paper Fast Graph Learning

tf-fsvd TensorFlow Implementation of Functional Singular Value Decomposition for paper Fast Graph Learning with Unique Optimal Solutions Cite If you f

Sami Abu-El-Haija 14 Nov 25, 2021