Unsupervised 3D Human Mesh Recovery from Noisy Point Clouds

Overview

Unsupervised 3D Human Mesh Recovery from Noisy Point Clouds

Xinxin Zuo, Sen Wang, Minglun Gong, Li Cheng

report

Prerequisites

We have tested the code on Ubuntu 18.04/20.04 with CUDA 10.2

Installation

First you have to make sure that you have all dependencies in place. The simplest way to do is to use the anaconda.

You can create an anaconda environment called fit3d using

conda env create -f environment.yaml
conda activate fit3d

Download SMPL models

Download SMPL Female and Male and SMPL Netural, and rename the files and extract them to /smpl_models/smpl/, eventually, the /smpl_models folder should have the following structure:

smpl_models
 └-- smpl
 	└-- SMPL_FEMALE.pkl
 	└-- SMPL_MALE.pkl
 	└-- SMPL_NEUTRAL.pkl

Download pre-trained models

  1. Download two weights (point cloud and depth) from: Point Cloud and Depth
  2. Put the downloaded weights in /pretrained/

Demo

Demo for whole point cloud

python generate_pt.py --filename ./demo/demo_pt/00010805.ply --gender female

Demo for depth/partial point cloud

python generate_depth.py --filename ./demo/demo_depth/shortshort_flying_eagle.000075_depth.ply --gender male

Input instruction

The input point cloud or depth's head should correspond the Y-axis. The X-Z rotation is acceptable.

Citation

If you find this project useful for your research, please consider citing:

@article{zuo2021unsupervised,
  title={Unsupervised 3D Human Mesh Recovery from Noisy Point Clouds},
  author={Zuo, Xinxin and Wang, Sen and Gong, Minglun and Cheng, Li},
  journal={arXiv preprint arXiv:2107.07539},
  year={2021}
}

References

We indicate if a function or script is borrowed externally inside each file. Here are some great resources we benefit:

Comments
  • Error: from pytorch3d import _C

    Error: from pytorch3d import _C

    Hi, I really appreciate your greate work!

    An error occurred when i run the "generate_depth.py", how do i solve it?

    : ~/unsupervised3dhuman-main$ python generate_depth.py --filename ./demo/demo_depth/shortshort_flying_eagle.000075_depth.ply --gender male Traceback (most recent call last): File "generate_depth.py", line 18, in from src.utils import index_points, farthest_point_sample File "/home/iot/PycharmProjects/unsupervised3dhuman-main/src/utils.py", line 7, in from pytorch3d import _C ImportError: libc10_cuda.so: cannot open shared object file: No such file or directory

    I create an anaconda environment according your "environment.yaml", the system is ubuntu18.04 and cuda10.2

    Thank you!

    opened by jiangSeu 14
  • Troubles in predicting pose

    Troubles in predicting pose

    Hi.

    I'd like to start thanking you, because I was looking for solutions on how to fit a point cloud for animation reasons, and your work is the best I could find on the topic, it is very close to what I'm trying to do and you published some code, so thank you so much for your effort.

    Sorry to disturb you, but I was experimenting with your algorithm on body scans, and I came across some issues. I am talking about body facing direction, legs crossing and body parts compenetration.

    I see that in previous issues you already solved these issues, but it is not working for me.

    I added the row -- pred_pose[0, 1] = (pred_pose[0, 1] + 1.57).unsqueeze(0).float() after line 100, but it is still not working I suspected the reason could be in the fact that my body is oriented along a different axis, but the model actually predicts the orientation correctly, it just places the body rotated 180 degrees (face towards the back of the original point cloud). This creates a pose estimation that is completely wrong. I am working on point clouds of various people in a slight A-pose, so I guess it should be a very easy task, so it is the only explanation for those weird poses generated.

    Any suggestion for this? Should I apply rotations to my original point cloud in advance? Am I doing something wrong? I was trying to make your code work before trying and training my own solution.

    About leg crossing, I saw you applied another correction, but the issue is in a language I don't understand, so it is difficult for me to understand if it has been already patched, or where should I change the code.

    I don't know if you are still working on this, and if you can help me, but I would be very grateful if you could find a minute to drop me some hints

    Thank you in advance for your consideration and your help

    Have a nice day

    opened by GiBucci 6
  • 关于扫描数据的测试

    关于扫描数据的测试

    你好,请原谅我英文不太好,怕表达的不清晰,就用中文说了。 我测试了您的网络。用的是FAUST的扫描模型。 请问是需要对这里的扫描模型进行预处理么?我直接用的话,得到的结果好像有些差,还是说我的环境有什么问题,我直接用的readme上的提示做的,创建的虚拟环境。 这是我的结果 image image image image 分别对应的scans中的096和035. 希望能得到您的帮助,不胜感激

    opened by qiji77 6
  • Depth demo

    Depth demo

    Hi,

    I ran the demo you provdied for fitting the SMPL model onto a partial scan:

    python generate_depth.py --filename ./demo/demo_depth/shortshort_flying_eagle.000075_depth.ply --gender male

    but the surface_EM_depth minimization outpus NaN s for the demo example you provided. Does the code work for you?

    Thanks in advance, David

    opened by DavidBoja 2
  • Dataset specifics

    Dataset specifics

    Hi,

    could you specify which subjects and frames did you use to evaluate the methods on the CAPE and CMU datasets (Table 1 and 2 from the paper) ?

    Thank you in advance.

    opened by DavidBoja 1
  • How to convert depth image to ply file

    How to convert depth image to ply file

    Hi, Wang sen, I am not very familiar with point cloud processing, I don't know how to convert depth image with camera intrinsic parameters to ply file, could you help me, thank you.

    opened by LinyeLi60 1
Owner
null
Mesh Graphormer is a new transformer-based method for human pose and mesh reconsruction from an input image

MeshGraphormer ✨ ✨ This is our research code of Mesh Graphormer. Mesh Graphormer is a new transformer-based method for human pose and mesh reconsructi

Microsoft 251 Jan 8, 2023
Code for "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds", CVPR 2021

PV-RAFT This repository contains the PyTorch implementation for paper "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clou

Yi Wei 43 Dec 5, 2022
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (CVPR 2022, Oral)

Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (CVPR 2022, Oral) This is the official implementat

Yifan Zhang 259 Dec 25, 2022
This repo is a PyTorch implementation for Paper "Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds"

Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds This repository is a PyTorch implementation for paper: Uns

Kaizhi Yang 42 Dec 9, 2022
A repo that contains all the mesh keys needed for mesh backend, along with a code example of how to use them in python

Mesh-Keys A repo that contains all the mesh keys needed for mesh backend, along with a code example of how to use them in python Have been seeing alot

Joseph 53 Dec 13, 2022
CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes Implementation of CoSMA: Convolutional Semi-Regular Mesh Autoencoder arXiv p

Fraunhofer SCAI 10 Oct 11, 2022
Given a 2D triangle mesh, we could randomly generate cloud points that fill in the triangle mesh

generate_cloud_points Given a 2D triangle mesh, we could randomly generate cloud points that fill in the triangle mesh. Run python disp_mesh.py Or you

Peng Yu 2 Dec 24, 2021
AI Face Mesh: This is a simple face mesh detection program based on Artificial intelligence.

AI Face Mesh: This is a simple face mesh detection program based on Artificial Intelligence which made with Python. It's able to detect 468 different

Md. Rakibul Islam 1 Jan 13, 2022
This project is the official implementation of our accepted ICLR 2021 paper BiPointNet: Binary Neural Network for Point Clouds.

BiPointNet: Binary Neural Network for Point Clouds Created by Haotong Qin, Zhongang Cai, Mingyuan Zhang, Yifu Ding, Haiyu Zhao, Shuai Yi, Xianglong Li

Haotong Qin 59 Dec 17, 2022
(CVPR 2021) PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds

PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds by Mutian Xu*, Runyu Ding*, Hengshuang Zhao, and Xiaojuan Qi. Int

CVMI Lab 228 Dec 25, 2022
《A-CNN: Annularly Convolutional Neural Networks on Point Clouds》(2019)

A-CNN: Annularly Convolutional Neural Networks on Point Clouds Created by Artem Komarichev, Zichun Zhong, Jing Hua from Department of Computer Science

Artёm Komarichev 44 Feb 24, 2022
Official PyTorch implementation of CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds

CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds Introduction This is the official PyTorch implementation of o

Yijia Weng 96 Dec 7, 2022
(CVPR 2021) Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds

BRNet Introduction This is a release of the code of our paper Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds,

null 86 Oct 5, 2022
Self-Supervised Learning for Domain Adaptation on Point-Clouds

Self-Supervised Learning for Domain Adaptation on Point-Clouds Introduction Self-supervised learning (SSL) allows to learn useful representations from

Idan Achituve 66 Dec 20, 2022
Rendering Point Clouds with Compute Shaders

Compute Shader Based Point Cloud Rendering This repository contains the source code to our techreport: Rendering Point Clouds with Compute Shaders and

Markus Schütz 460 Jan 5, 2023
This is a package for LiDARTag, described in paper: LiDARTag: A Real-Time Fiducial Tag System for Point Clouds

LiDARTag Overview This is a package for LiDARTag, described in paper: LiDARTag: A Real-Time Fiducial Tag System for Point Clouds (PDF)(arXiv). This wo

University of Michigan Dynamic Legged Locomotion Robotics Lab 159 Dec 21, 2022
Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving

SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving Abstract In this paper, we introduce SalsaNext f

null 308 Jan 4, 2023
Code for "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" @ICRA2021

CloudAAE This is an tensorflow implementation of "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" Files log:

Gee 35 Nov 14, 2022
Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather

LiDAR fog simulation Created by Martin Hahner at the Computer Vision Lab of ETH Zurich. This is the official code release of the paper Fog Simulation

Martin Hahner 110 Dec 30, 2022