Subdivision-based Mesh Convolutional Networks

Overview

Subdivision-based Mesh Convolutional Networks

The official implementation of SubdivNet in our paper,

Subdivion-based Mesh Convolutional Networks

teaser

Requirements

  • python3.7+
  • CUDA 10.1+
  • Jittor

To install python requirements:

pip install -r requirements.txt

Fetch Data

This repo provides training scripts for classification and segementation, on the following datasets,

  • shrec11-split10
  • shrec11-split16
  • cubes
  • manifold40 (based on ModelNet40)
  • humanbody
  • coseg-aliens

To download the preprocessed data, run

sh scripts/<DATASET_NAME>/get_data.sh

Manfold40 (before remeshed) can be downloaded via this link.

Training

To train the model(s) in the paper, run this command:

sh scripts/<DATASET_NAME>/train.sh

To speed up training, you can use multiple gpus. First install OpenMPI:

sudo apt install openmpi-bin openmpi-common libopenmpi-dev

Then run the following command,

CUDA_VISIBLE_DEVICES="2,3" mpirun -np 2 sh scripts/<DATASET_NAME>/train.sh

Evaluation

To evaluate the model on a dataset, run:

sh scripts/<DATASET_NAME>/test.sh

The pretrained weights are provided. Run the following command to download them.

sh scripts/<DATASET_NAME>/get_pretrained.sh

Visualize

After testing the segmentation network, there will be colored shapes in a results directory. Use your favorite 3D viewer to check them.

Apply to your own data

To create your own data with subdivision sequence connectivity, you may use our provided tool that implements the MAPS algorithm. You may also refer to NeuralSubdivision, as they also provide a MATLAB scripts for remeshing.

To run our implemented MAPS algorithm, first install the following python dependecies,

triangle
pymeshlab
shapely
sortedcollections
networkx
rtree

Then run datagen_maps.py to remesh your meshes.

Cite

Please cite our paper if you use this code in your own work:

@misc{hu2021subdivisionbased,
      title={Subdivision-Based Mesh Convolution Networks}, 
      author={Shi-Min Hu and Zheng-Ning Liu and Meng-Hao Guo and Jun-Xiong Cai and Jiahui Huang and Tai-Jiang Mu and Ralph R. Martin},
      year={2021},
      eprint={2106.02285},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
Comments
  • 关于数据集训练的一些问题

    关于数据集训练的一些问题

    @lzhengning 您好!

    感谢您能够开源这么优秀的项目!

    目前我在尝试使用自己的数据集进行训练,但是现在遇到一些问题: 1、数据集预处理部分,那个json文件中的参数我没搞明白,我看过了您在#9 中的回答,那个sub_labels和raw_to_sub的数据是如何获取的,trimesh.proximity.closest_point中的输入是一个mesh数据和一个points,这个points应该选择哪些点的坐标?mesh的顶点吗? 2、在打标签时,应该在原始数据上进行还是在进行remesh后的mesh中进行? 3、我目前需要训练的mesh数据每个都包含了10w个左右的三角面片,请问还需要运行MAPS算法吗?如果需要,是否应该先对mesh进行简化后再运行该算法?

    期待您的回答,谢谢!

    opened by ZYB-1997 14
  • How many gpus do you utilize?

    How many gpus do you utilize?

    ”name: manifold40 Train 0: 0%| | 0/2049 [00:00<?, ?it/s] Compiling Operators(1/1) used: 19.2s eta: 0s

    Compiling Operators(1/1) used: 13s eta: 0s

    Compiling Operators(1/1) used: 13s eta: 0s

    Compiling Operators(1/1) used: 13.8s eta: 0s

    Compiling Operators(56/56) used: 78.5s eta: 0s

    Compiling Operators(44/44) used: 60.4s eta: 0s

    Compiling Operators(1/1) used: 16s eta: 0s Train 0: 0%| | 1/2049 [03:56<134:34:05, 236.55s/it] Compiling Operators(2/2) used: 19.3s eta: 0s

    Compiling Operators(1/1) used: 14.4s eta: 0s Train 0: 1%|▋ | 21/2049 [04:52<1:14:44, 2.21s/it][w 0624 09:21:26.817385 20 cudnn_conv_Tx:float32__Ty:float32__Tw:float32__XFORMAT:abcd__WFORMAT:oihw__YFORMAT:abcd__J...hash:798cb5ed49dadaa2_op.cc:200] forward_ algorithm cache is full [w 0624 09:21:27.423196 20 cudnn_conv_backward_w_Tx:float32__Ty:float32__Tw:float32__XFORMAT:abcd__WFORMAT:oihw__YFOR...hash:c41e3d43aa5d4cf7_op.cc:199] backward w algorithm cache is full Train 0: 1%|▊ | 24/2049 [04:56<56:14, 1.67s/it][w 0624 09:21:31.279631 20 cudnn_conv_backward_x_Tx:float32__Ty:float32__Tw:float32__XFORMAT:abcd__WFORMAT:oihw__YFOR...hash:74f24b7a5fa4fe17_op.cc:201] backward x algorithm cache is full Train 0: 65%|███████████████████████████████████████████▌ | 1332/2049 [1:21:43<51:18, 4.29s/it] “ The training process consume too much time. I utilize 1 gpu (12G) to train. An hour later, Train 0 is not over. I noticed that you mentioned that using multiple gpus could speed up the trainning, so I want to know how many gpus do you utilize and how long is the training time?

    Thank you very much!

    opened by liang3588 12
  • Segmentation on my own dataset

    Segmentation on my own dataset

    Hi @lzhengning, Thanks for releasing this code.

    I would like to test your pretrained segmentation model on .obj meshes without groundtruth, do you think it is easily achievable ? Is there a particular tool you use to label the meshes and get the .json files present in your repository ?

    Kind regards.

    opened by goyallon 7
  • Questions about your results?

    Questions about your results?

    The results of PD-MeshNet in your paper are very different from the results in the original paper. I also notice your answer to question 15. But the metric of PD-MeshNet is on the primal nodes (on faces), which means that the metric is strict. Therefore, I'm curious about the reason. Thanks! Have a nice day!

    opened by lidan233 6
  • AssertionError assert self.F == self.Fs.max().data[0]

    AssertionError assert self.F == self.Fs.max().data[0]

    你好 我在进行manifold数据集上的分类任务训练时,遇到了下面的错误

    name:  manifold40
    Train 0:   0%|                                                                                                                                                                           | 0/2049 [00:01<?, ?it/s]
    Traceback (most recent call last):
      File "train_cls.py", line 184, in <module>
        train(net, optim, train_dataset, writer, epoch)
      File "train_cls.py", line 33, in train
        mesh_tensor = to_mesh_tensor(meshes)
      File "/mnt/data/wy/SubdivNet/subdivnet/utils.py", line 28, in to_mesh_tensor
        jt.int32(meshes['Fs']))
      File "/mnt/data/wy/SubdivNet/subdivnet/mesh_tensor.py", line 35, in __init__
        assert self.F == self.Fs.max().data[0]
    AssertionError
    

    我是在ubuntu18.04上用conda搭建环境的,jittor版本1.3.0.14,期待您的答复。

    opened by flyKite1998 5
  • 无法复现论文结果,一些预训练分类模型准确率为0

    无法复现论文结果,一些预训练分类模型准确率为0

    使用了cubesmanifold40shrec11-split10shrec11-split16的预训练模型,分类结果:

    cubes

    name:  Cubes
    Test 0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 103/103 [01:09<00:00,  1.48it/s]
    test acc =  0.00015174506828528073
    test acc [voted] =  0.0
    

    manifold40

    name:  Manifold40
    Test 0:   5%|███████                                                                                                                          | 112/2057 [00:11<04:29,  7.22it/s]
    [w 0723 15:28:27.917150 76 cudnn_conv_Tx:float32__Ty:float32__Tw:float32__XFORMAT:abcd__WFORMAT:oihw__YFORMAT:abcd__J...hash:798cb5ed49dadaa2_op.cc:200] forward_ algorithm cache is full
    Test 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2057/2057 [03:12<00:00, 10.67it/s]
    test acc =  0.037560777957860614
    test acc [voted] =  0.03727714748784441
    

    shrec11-split10

    name:  shrec11-split10
    Test 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 99/99 [00:48<00:00,  2.04it/s]
    test acc =  0.0
    test acc [voted] =  0.0
    

    shrec11-split16

    name:  shrec11-split16
    Test 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:28<00:00,  1.43it/s]
    test acc =  0.0
    test acc [voted] =  0.0
    
    opened by zhenhao-huang 5
  • Cannot run the test shell script

    Cannot run the test shell script

    Hello. Thanks for your work. However, when I try to run the test script of coseg-alien, it gives me an error message like this:

    name: coseg-alien
    0:   0%|                    | 0/37 [00:00<?, ?it/s]/home/xxx/.cache/jittor/default/g++/jit/_opkey0:array_T:int32__JIT:1__JIT_cuda:1__index_t:int32___opkey1:broadcast_to_Tx:int32__DI...hash:bc2f95c82b48131a_op.cc(40): error: calling a constexpr __host__ function("floor") from a __global__ function("func_bc2f95c82b48131a_0") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
    
    1 error detected in the compilation of "/home/xxx/.cache/jittor/default/g++/jit/_opkey0:array_T:int32__JIT:1__JIT_cuda:1__index_t:int32___opkey1:broadcast_to_Tx:int32__DI...hash:bc2f95c82b48131a_op.cc".
    [e 0721 14:21:37.874137 48:C15 parallel_compiler.cc:261] [Error] source file location: /home/xxx/.cache/jittor/default/g++/jit/_opkey0:array_T:int32__JIT:1__JIT_cuda:1__index_t:int32___opkey1:broadcast_to_Tx:int32__DI...hash:bc2f95c82b48131a_op.cc
    [e 0721 14:21:37.874176 48:C15 parallel_compiler.cc:264] Compile fused operator(18/56) failed: [Op(0x5561911a3de0:0:0:1:i0:o1:s0,array->0x5561911a3e80),Op(0x5561911a2360:0:0:1:i1:o1:s0,broadcast_to->0x5561911a2410),Op(0x5561911a37f0:0:0:1:i2:o1:s0,binary.mod->0x5561911a30a0),] 
    
    Reason: [f 0721 14:21:37.873882 48:C15 log.cc:387] Check failed ret(256) == 0(0) Run cmd failed: cd /home/xxx/.cache/jittor/default/g++ && /home/xxx/.cache/jittor/jtcuda/cuda11.2_cudnn8_linux/bin/nvcc '/home/xxx/.cache/jittor/default/g++/jit/_opkey0:array_T:int32__JIT:1__JIT_cuda:1__index_t:int32___opkey1:broadcast_to_Tx:int32__DI...hash:bc2f95c82b48131a_op.cc'     -std=c++14 -Xcompiler -fPIC  -Xcompiler -march=native  -Xcompiler -fdiagnostics-color=always  -I/home/xxx/anaconda3/envs/subdivnet/lib/python3.7/site-packages/jittor/src -I/home/xxx/anaconda3/envs/subdivnet/include/python3.7m -I/home/xxx/anaconda3/envs/subdivnet/include/python3.7m -DHAS_CUDA -I'/home/xxx/.cache/jittor/jtcuda/cuda11.2_cudnn8_linux/include' -I'/home/xxx/anaconda3/envs/subdivnet/lib/python3.7/site-packages/jittor/extern/cuda/inc'  -lstdc++ -ldl -shared  -x cu --cudart=shared -ccbin='/usr/bin/g++' --use_fast_math  -w  -I'/home/xxx/anaconda3/envs/subdivnet/lib/python3.7/site-packages/jittor/extern/cuda/inc'  -arch=compute_61  -code=sm_61  -o '/home/xxx/.cache/jittor/default/g++/jit/_opkey0:array_T:int32__JIT:1__JIT_cuda:1__index_t:int32___opkey1:broadcast_to_Tx:int32__DI...hash:bc2f95c82b48131a_op.so'
    0:   0%|                    | 0/37 [00:02<?, ?it/s]
    Traceback (most recent call last):
      File "train_seg.py", line 162, in <module>
        test(net, test_dataset, writer, 0, args)
      File "/home/xxx/anaconda3/envs/subdivnet/lib/python3.7/site-packages/jittor/__init__.py", line 257, in inner
        ret = func(*args, **kw)
      File "train_seg.py", line 64, in test
        preds = np.argmax(outputs.data, axis=1)
    RuntimeError: Wrong inputs arguments, Please refer to examples(help(jt.data)).
    
    Types of your inputs are:
     self   = Var,
    
    The function declarations are:
     inline DataView data()
    
    Failed reason:[f 0721 14:21:38.068584 96 parallel_compiler.cc:316] Error happend during compilation, see error above.
    

    Any ideas on why this happens? I have downloaded the data of coseg-alien via the shell script provided.

    opened by unw9527 5
  • Could you please provide datasets with subdivision connectivity

    Could you please provide datasets with subdivision connectivity

    I found the datasets download via "sh scripts/<DATASET_NAME>/get_data.sh" are original datasets without the subdivision connectivity. Moreover, it is very hard to remesh them by datagen_maps.py for the remeshing is always interrupted by some problems in the program, especially for the Manifold40.

    Therefore, could you please provide the remeshed datasets with subdivision connectivity? Thank you very much!

    opened by liang3588 5
  • cannot reproduce results on Manifold40

    cannot reproduce results on Manifold40

    Hi, I'm trying to reproduce your results on Manifold40.

    I downloaded the dataset with the command sh scripts/manifold40/get_data.sh, which downloads the file Manifold40-MAPS-96-3.zip. I noticed in your instructions that it is possible to download a different version of Manifold40 "before remeshing". Which is the correct version to use for evaluation?

    I noticed that the version in Manifold40-MAPS-96-3.zip contains approximatively 10 times the number of shapes contained in the original ModelNet40, why is that?

    I run the command sh scripts/manifold40/get_pretrained.sh to obtain the checkpoint for the model and then I run the command sh scripts/manifold40/test.sh to perform evaluation. The evaluation runs fine but I get a very low accuracy. Do you have any idea of what I could have missed?

    Thanks in advance, Luca.

    opened by lykius 4
  • Data preprocessing

    Data preprocessing

    I'm trying to use your script datagen_maps.py to preprocess some meshes in order to feed them to SubDivNet. The meshes are obtained with marching cubes and I tried different resolutions to get meshes with different number of vertices/faces. The preprocessing fails in the decimate() function here, without any meaningful error (at least that I can understand).

    Do you have any suggestion? Is there any special requirement for the meshes in order to be able to preprocess them?

    Thanks in advance!

    opened by lykius 3
  • question concerning labeling the data

    question concerning labeling the data

    Hello, thanks for your great work and offered code.

    i'm working on a school project and i want to use the SUBDIVNET code to train my data, but there are some points i did not understand..

    in each .obj file there are 3 indexes:

    • raw_labels: each item here represents a label of a face in the original data.
    • sub_labels: each item here represents a label of a face in the data after remeshing it (after applying datagen_maps.py on it )
    • raw_to_sub: this i don't understand what does it represent, and what is its role

    another thing i didn't understand: length of raw_labels and sub_labels is 13776 but number of faces in each object is 16384. so if raw_labels and sub_labels are representing the label of each face why they don't have same length as faces.

    so can you please explain to me how can i preprocess step by step my data to before feeding it to the subdivnet? how can i make each object have the exact number of faces as in your input data (16384) and how to label my data, and if you can recommend me a tool i can use!

    thank you in advance

    opened by hamzam0n 2
  • Questions about input data for segmentation

    Questions about input data for segmentation

    Thanks for sharing the code of your nice work! I've been trying to apply the framework to my own dataset for segmentation. As I understood, the dataroot contains the folders 'train', 'test' and 'raw'. For each sample, there are three files: *.obj, *.json (labels), and the *.off (in 'raw'). What is unclear to me are:

    1. What are the data in the 'raw' folder? I expect the raw data to be *.obj. What is the purpose of saving the meshes in .off instead of .obj? The remeshing algorithms (MAPS and neural subdivision) have the finest mesh as input and downsample the meshes. However, the meshes in the 'raw' folders have a lower resolution than those in the 'train' and 'test' folders.
    2. How do you obtain the labels (*.json)?

    Thank you!

    opened by whsu2s 1
  • About one sample in Manifold40

    About one sample in Manifold40

    Hi, thanks for you great paper and code! When I want to make use of the new dataset Manifold40 you proposed from tsinghua cloud to training my own model, I found that there was a mere mistake in only one sample that is 'bookshelf_0099.obj'. The content of the version I downloaded is:

    Object res.obj

    Vertices: 245

    Faces: 842

    with 842 faces, not 500, so I remeshed it in meshlab to make the code works.

    opened by MyEternalMoon 2
Owner
Zheng-Ning Liu
Zheng-Ning Liu
Mesh Graphormer is a new transformer-based method for human pose and mesh reconsruction from an input image

MeshGraphormer ✨ ✨ This is our research code of Mesh Graphormer. Mesh Graphormer is a new transformer-based method for human pose and mesh reconsructi

Microsoft 251 Jan 8, 2023
AI Face Mesh: This is a simple face mesh detection program based on Artificial intelligence.

AI Face Mesh: This is a simple face mesh detection program based on Artificial Intelligence which made with Python. It's able to detect 468 different

Md. Rakibul Islam 1 Jan 13, 2022
A repo that contains all the mesh keys needed for mesh backend, along with a code example of how to use them in python

Mesh-Keys A repo that contains all the mesh keys needed for mesh backend, along with a code example of how to use them in python Have been seeing alot

Joseph 53 Dec 13, 2022
Given a 2D triangle mesh, we could randomly generate cloud points that fill in the triangle mesh

generate_cloud_points Given a 2D triangle mesh, we could randomly generate cloud points that fill in the triangle mesh. Run python disp_mesh.py Or you

Peng Yu 2 Dec 24, 2021
This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of Coordinate Independent Convolutional Networks.

Orientation independent Möbius CNNs This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of

Maurice Weiler 59 Dec 9, 2022
An Efficient Implementation of Analytic Mesh Algorithm for 3D Iso-surface Extraction from Neural Networks

AnalyticMesh Analytic Marching is an exact meshing solution from neural networks. Compared to standard methods, it completely avoids geometric and top

Karbo 45 Dec 21, 2022
Range Image-based LiDAR Localization for Autonomous Vehicles Using Mesh Maps

Range Image-based 3D LiDAR Localization This repo contains the code for our ICRA2021 paper: Range Image-based LiDAR Localization for Autonomous Vehicl

Photogrammetry & Robotics Bonn 208 Dec 15, 2022
A benchmark dataset for mesh multi-label-classification based on cube engravings introduced in MeshCNN

Double Cube Engravings This script creates a dataset for multi-label mesh clasification, with an intentionally difficult setup for point cloud classif

Yotam Erel 1 Nov 30, 2021
Search and filter videos based on objects that appear in them using convolutional neural networks

Thingscoop: Utility for searching and filtering videos based on their content Description Thingscoop is a command-line utility for analyzing videos se

Anastasis Germanidis 354 Dec 4, 2022
PyTorch implementation of convolutional neural networks-based text-to-speech synthesis models

Deepvoice3_pytorch PyTorch implementation of convolutional networks-based text-to-speech synthesis models: arXiv:1710.07654: Deep Voice 3: Scaling Tex

Ryuichi Yamamoto 1.8k Jan 8, 2023
This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning].

CG3 This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning]. R

null 12 Oct 28, 2022
Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition in CVPR19

2s-AGCN Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition in CVPR19 Note PyTorch version should be 0.3! For PyTor

LShi 547 Dec 26, 2022
Spatial Temporal Graph Convolutional Networks (ST-GCN) for Skeleton-Based Action Recognition in PyTorch

Reminder ST-GCN has transferred to MMSkeleton, and keep on developing as an flexible open source toolbox for skeleton-based human understanding. You a

sijie yan 1.1k Dec 25, 2022
Codes for TIM2021 paper "Anchor-Based Spatio-Temporal Attention 3-D Convolutional Networks for Dynamic 3-D Point Cloud Sequences"

Codes for TIM2021 paper "Anchor-Based Spatio-Temporal Attention 3-D Convolutional Networks for Dynamic 3-D Point Cloud Sequences"

Intelligent Robotics and Machine Vision Lab 4 Jul 19, 2022
The official implementation of NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation [ICLR-2021]. https://arxiv.org/pdf/2101.12378.pdf

NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation [ICLR-2021] Release Notes The offical PyTorch implementation of NeMo, p

Angtian Wang 76 Nov 23, 2022
[ECCV 2020] Reimplementation of 3DDFAv2, including face mesh, head pose, landmarks, and more.

Stable Head Pose Estimation and Landmark Regression via 3D Dense Face Reconstruction Reimplementation of (ECCV 2020) Towards Fast, Accurate and Stable

Remilia Scarlet 221 Dec 30, 2022
MediaPipeのPythonパッケージのサンプルです。2020/12/11時点でPython実装のある4機能(Hands、Pose、Face Mesh、Holistic)について用意しています。

mediapipe-python-sample MediaPipeのPythonパッケージのサンプルです。 2020/12/11時点でPython実装のある以下4機能について用意しています。 Hands Pose Face Mesh Holistic Requirement mediapipe 0.

KazuhitoTakahashi 217 Dec 12, 2022
Open-AI's DALL-E for large scale training in mesh-tensorflow.

DALL-E in Mesh-Tensorflow [WIP] Open-AI's DALL-E in Mesh-Tensorflow. If this is similarly efficient to GPT-Neo, this repo should be able to train mode

EleutherAI 432 Dec 16, 2022
Mesh TensorFlow: Model Parallelism Made Easier

Mesh TensorFlow - Model Parallelism Made Easier Introduction Mesh TensorFlow (mtf) is a language for distributed deep learning, capable of specifying

null 1.3k Dec 26, 2022