Diffusion Probabilistic Models for 3D Point Cloud Generation (CVPR 2021)

Overview

Diffusion Probabilistic Models for 3D Point Cloud Generation

teaser

[Paper] [Code]

The official code repository for our CVPR 2021 paper "Diffusion Probabilistic Models for 3D Point Cloud Generation".

Installation

[Option 1] Install via conda environment YAML file (CUDA 10.1).

# Create the environment
conda env create -f env.yml
# Activate the environment
conda activate dpm-pc-gen

[Option 2] Or you may setup the environment manually (If you are using GPUs that only work with CUDA 11 or greater).

Our model only depends on the following commonly used packages, all of which can be installed via conda.

Package Version
PyTorch ≥ 1.6.0
h5py not specified (we used 4.61.1)
tqdm not specified
tensorboard not specified (we used 2.5.0)
numpy not specified (we used 1.20.2)
scipy not specified (we used 1.6.2)
scikit-learn not specified (we used 0.24.2)

About the EMD Metric

We have removed the EMD module due to GPU compatability issues. The legacy code can be found on the emd-cd branch.

If you have to compute the EMD score or compare our model with others, we strongly advise you to use your own code to compute the metrics. The generation and decoding results will be saved to the results folder after each test run.

Datasets and Pretrained Models

Datasets and pretrained models are available at: https://drive.google.com/drive/folders/1Su0hCuGFo1AGrNb_VMNnlF7qeQwKjfhZ

Training

# Train an auto-encoder
python train_ae.py 

# Train a generator
python train_gen.py

You may specify the value of arguments. Please find the available arguments in the script.

Note that --categories can take all (use all the categories in the dataset), airplane, chair (use a single category), or airplane,chair (use multiple categories, separated by commas).

Testing

# Test an auto-encoder
python test_ae.py --ckpt ./pretrained/AE_all.pt --categories all

# Test a generator
python test_gen.py --ckpt ./pretrained/GEN_airplane.pt --categories airplane

Citation

@inproceedings{luo2021diffusion,
  author = {Luo, Shitong and Hu, Wei},
  title = {Diffusion Probabilistic Models for 3D Point Cloud Generation},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2021}
}
Comments
  • 请问训练和测试都出现RuntimeError: CUDA error: invalid device function错误

    请问训练和测试都出现RuntimeError: CUDA error: invalid device function错误

    你好,请问训练和测试都出现如下错误,是否环境配置原因?

    EMD-CD: 0%| | 0/19 [00:00<?, ?it/s] Traceback (most recent call last): File "train_ae.py", line 203, in cd_loss = validate_loss(it) File "train_ae.py", line 169, in validate_loss metrics = EMD_CD(all_recons, all_refs, batch_size=args.val_batch_size, accelerated_cd=True) File "/userhome/point_cloud/diffusion-point-cloud-main/evaluation/evaluation_metrics.py", line 58, in EMD_CD cd_lst.append(dl.mean(dim=1) + dr.mean(dim=1)) RuntimeError: CUDA error: invalid device function Segmentation fault (core dumped)

    opened by Heroism502 10
  • 请问如下两个错误怎么解决:1、error in nnd Output: invalid device function。2、AssertionError: can only join a child process

    请问如下两个错误怎么解决:1、error in nnd Output: invalid device function。2、AssertionError: can only join a child process

    问题1: image

    问题2: Exception ignored in: <function _MultiProcessingDataLoaderIter.del at 0x7fb8cf914320> Traceback (most recent call last): File "/root/miniconda3/envs/dpm-pc-gen/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1101, in del self._shutdown_workers() File "/root/miniconda3/envs/dpm-pc-gen/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1075, in _shutdown_workers w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL) File "/root/miniconda3/envs/dpm-pc-gen/lib/python3.7/multiprocessing/process.py", line 138, in join assert self._parent_pid == os.getpid(), 'can only join a child process' AssertionError: can only join a child process

    opened by Heroism502 7
  • CUDA error

    CUDA error

    Hi folks,

    when training the model with "Car" shapes, the process is halted because of the following error

    Pairwise EMD-CD: 9%|########6 | 36/400 [00:00<00:03, 97.56it/s] Traceback (most recent call last): File "train_gen.py", line 218, in test(it) File "train_gen.py", line 185, in test results = compute_all_metrics(gen_pcs.to(args.device), ref_pcs.to(args.device), args.val_batch_size, accelerated_cd=True) File "/pointcloudsgan/code/diffusion-point-cloud/evaluation/evaluation_metrics.py", line 196, in compute_all_metrics ref_pcs, sample_pcs, batch_size, accelerated_cd=accelerated_cd) File "/pointcloudsgan/code/diffusion-point-cloud/evaluation/evaluation_metrics.py", line 110, in pairwise_EMD_CD emd_batch = emd_approx(sample_batch_exp, ref_batch) File "/pointcloudsgan/code/diffusion-point-cloud/evaluation/evaluation_metrics.py", line 21, in emd_approx emd = match_cost(sample, ref) # (B,) File "/pointcloudsgan/code/diffusion-point-cloud/evaluation/StructuralLosses/match_cost.py", line 22, in forward cost = MatchCost(seta, setb, match) RuntimeError: CUDA kernel failed : 700

    It is quite strange since we use the pretrained model to test airplane/chair generation and don't have any problems using Pairwise EMD-CD. Any insights?

    Our GPU is RTX 6000 with CUDA 11.0. StructuralLoss was also correctly compiled.

    Many thanks in advance.

    opened by titaniumrain 5
  • About the Drawing of 3D point cloud generation Images in the paper

    About the Drawing of 3D point cloud generation Images in the paper

    How to draw the image generated by the beautiful 3D Point Cloud Generation in this article, can anybody share the code of the drawing? Since i like it very much.

    opened by ZhouHaoWa 2
  • Question about training

    Question about training

    Hi! I am interested in your wonderful work!

    I have a question about training. In your training file "train_gen.py", you set the training iteration number to 'inf' and the training loss is around 60.5, also the grad is around 0.2 to 0.3. So I want to know when to stop straining and I cannot judge stop time just by the loss and grad. Also I want to know why the training loss almost ever decrease a lot.

    Looking forward to your response!

    Best.

    opened by yufeng9819 2
  • /usr/bin/ld: cannot find -lmake_pytorch collect2: error: ld returned 1 exit status error: command 'g++' failed with exit status 1

    /usr/bin/ld: cannot find -lmake_pytorch collect2: error: ld returned 1 exit status error: command 'g++' failed with exit status 1

    Hi,

    I have built "StructuralLoss" successfully. But for the evaluation_metrics.py, I found the problem that it can not import " from evaluation.StructuralLosses.StructuralLossesBackend import ApproxMatch, MatchCost, MatchCostGrad" in match_cost.py ( folder: pytorch_structural_losses) because "StructuralLossesBackend" is not created yet.

    So, I tried to go to "pytorch_structural_losses" folder and run the "setup.py" file using the following command:

    python setup.py build_ext

    But getting errors. ( I have attached a screenshot) . I tried both python 3.7 and 3.8 but getting the same errors.

    I tried multiple times but could not able to find it. Please let me know how to fix this issue?

    Screen Shot 2021-08-05 at 6 09 25 PM
    opened by InfyIT-fa 2
  • About visualization

    About visualization

    Hi, first thanks for your inspiring work! Point cloud rendering figures in your paper are so beautiful, such as these vivid shadows. How do you draw it? Using open3d, meshlab or other programmes?

    Thanks! image

    opened by StevenZzz07 2
  • Point cloud to mesh

    Point cloud to mesh

    Hello:) I got a point cloud, but I don't know how to wrap it in a mesh, I tried a lot of methods, but they all turn out too not pretty (example in the picture). Do you know how to improve the point cloud to get a better mesh shape? or maybe you know cool algorithms for getting a beautiful mesh shape? image

    opened by Siziff 1
  • a question about the `standard_normal_logprob` function

    a question about the `standard_normal_logprob` function

    Hi, I'm confused about the formula below: image I can understand the calculation of entropy by the formula below, image However, I cannot understand why the first item is calculated by standard_normal_logprob function, the function looks like a sum of lnf(x). how this function can correspond to the first item?

    Thank you

    opened by nicolgo 1
  • How to train

    How to train

    Dear Sir: Thanks for your sharing your elegant work. I am still confused about how to train this model on ShapeNet, even after reading your paper. Could you provide your training code for my better understanding

    opened by Chenfeng1271 1
  • Huggingface Demo I implemented

    Huggingface Demo I implemented

    Guys , I implemented your awesome work to HuggingFace as a demo . People can try your work . You can reach me for any edit

    https://huggingface.co/spaces/SerdarHelli/diffusion-point-cloud

    opened by SerdarHelli 1
  • Quetion about parameters

    Quetion about parameters

    Hi! Thanks for your wonderful work and I am following your work.

    I find that diffusion model used in image areas usually follow settings in Ho et al.(2020) setting num steps to 1000 and beta T to 0.01/0.02 but in your paper, num steps fixed to 200 and beta T set to 0.05. I just want to know how to finally get those parameters (or just try all kinds of cases) since I can not get reasonable result after changing those parameters some times when I apply your work to other point cloud problem.

    Looking forward to your reply! Thanks a lot.

    opened by yufeng9819 1
  • Number of iteration to train?

    Number of iteration to train?

    Hi Zhitong, thanks for releasing the code! I am wondering how many training iterations is needed for different model/data to reproduce the results in the paper?

    Best,

    opened by ZENGXH 1
Owner
Shitong Luo
Undergraduate @ PKU
Shitong Luo
Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021)

Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021) An efficient PyTorch library for Point Cloud Completion.

Microsoft 119 Jan 2, 2023
Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

null 75 Nov 24, 2022
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (ICCV 2021 Oral)

ILVR + ADM This is the implementation of ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (ICCV 2021 Oral). This repository is h

Jooyoung Choi 225 Dec 28, 2022
A denoising diffusion probabilistic model (DDPM) tailored for conditional generation of protein distograms

Denoising Diffusion Probabilistic Model for Proteins Implementation of Denoising Diffusion Probabilistic Model in Pytorch. It is a new approach to gen

Phil Wang 108 Nov 23, 2022
PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud, CVPR 2019.

PointRCNN PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud Code release for the paper PointRCNN:3D Object Proposal Generation a

Shaoshuai Shi 1.5k Dec 27, 2022
Official PyTorch implementation for FastDPM, a fast sampling algorithm for diffusion probabilistic models

Official PyTorch implementation for "On Fast Sampling of Diffusion Probabilistic Models". FastDPM generation on CIFAR-10, CelebA, and LSUN datasets. S

Zhifeng Kong 68 Dec 26, 2022
Denoising Diffusion Probabilistic Models

Denoising Diffusion Probabilistic Models This repo contains code for DDPM training. Based on Denoising Diffusion Probabilistic Models, Improved Denois

Alexander Markov 7 Dec 15, 2022
Implementation of Retrieval-Augmented Denoising Diffusion Probabilistic Models in Pytorch

Retrieval-Augmented Denoising Diffusion Probabilistic Models (wip) Implementation of Retrieval-Augmented Denoising Diffusion Probabilistic Models in P

Phil Wang 55 Jan 1, 2023
[ICCV 2021 Oral] SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer

This repository contains the source code for the paper SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer (ICCV 2021 Oral). The project page is here.

AllenXiang 65 Dec 26, 2022
NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling @ INTERSPEECH 2021 Accepted

NU-Wave — Official PyTorch Implementation NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling Junhyeok Lee, Seungu Han @ MINDsLab Inc

MINDs Lab 242 Dec 23, 2022
McGill Physics Hackathon 2021: Reaction-Diffusion Models for the Generation of Biological Patterns

DiffuseAnimals: Reaction-Diffusion Models for the Generation of Biological Patterns Introduction Reaction-diffusion equations can be utilized in order

Austin Szuminsky 2 Mar 7, 2022
Code for "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds", CVPR 2021

PV-RAFT This repository contains the PyTorch implementation for paper "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clou

Yi Wei 43 Dec 5, 2022
[CVPR 2021] Few-shot 3D Point Cloud Semantic Segmentation

Few-shot 3D Point Cloud Semantic Segmentation Created by Na Zhao from National University of Singapore Introduction This repository contains the PyTor

null 117 Dec 27, 2022
Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 90 Dec 29, 2022
Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences Introduction Point cloud sequences are irregular and unordered in the spatial dimen

Hehe Fan 63 Dec 9, 2022
Implementation of the "Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos" paper.

Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos Introduction Point cloud videos exhibit irregularities and lack of or

Hehe Fan 101 Dec 29, 2022
Synthetic LiDAR sequential point cloud dataset with point-wise annotations

SynLiDAR dataset: Learning From Synthetic LiDAR Sequential Point Cloud This is official repository of the SynLiDAR dataset. For technical details, ple

null 78 Dec 27, 2022
NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling

NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling For Official repo of NU-Wave: A Diffusion Probabilistic Model for Neural Audio Up

Rishikesh (ऋषिकेश) 38 Oct 11, 2022
Pytorch implementation of "Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech"

GradTTS Unofficial Pytorch implementation of "Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech" (arxiv) About this repo This is an unoffic

HeyangXue1997 103 Dec 23, 2022