The code for paper "Learning Implicit Fields for Generative Shape Modeling".

Overview

implicit-decoder

The tensorflow code for paper "Learning Implicit Fields for Generative Shape Modeling", Zhiqin Chen, Hao (Richard) Zhang.

Project page | Paper

Improved TensorFlow1 implementation

Improved PyTorch implementation

Update

We have an improved implementation here, where we trained one model on the 13 ShapeNet categories.

We have a PyTorch implementation here.

Introduction

We advocate the use of implicit fields for learning generative models of shapes and introduce an implicit field decoder, called IM-NET, for shape generation, aimed at improving the visual quality of the generated shapes. An implicit field assigns a value to each point in 3D space, so that a shape can be extracted as an iso-surface. IM-NET is trained to perform this assignment by means of a binary classifier. Specifically, it takes a point coordinate, along with a feature vector encoding a shape, and outputs a value which indicates whether the point is outside the shape or not. By replacing conventional decoders by our implicit decoder for representation learning (via IM-AE) and shape generation (via IM-GAN), we demonstrate superior results for tasks such as generative shape modeling, interpolation, and single-view 3D reconstruction, particularly in terms of visual quality.

Citation

If you find our work useful in your research, please consider citing:

@article{chen2018implicit_decoder,
  title={Learning Implicit Fields for Generative Shape Modeling},
  author={Chen, Zhiqin and Zhang, Hao},
  journal={Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2019}
}

Dependencies

Requirements:

Our code has been tested with Python 3.5, TensorFlow 1.8.0, CUDA 9.1 and cuDNN 7.0 on Ubuntu 16.04 and Windows 10.

Datasets and Pre-trained weights

The original voxel models and rendered views are from HSP. Since our network takes point-value pairs, the voxel models require further sampling. The sampling method can be found in our project page.

We provide the ready-to-use datasets in hdf5 format, together with our pre-trained network weights. The weights for IM-GAN is the ones we used in our demo video. The weights for IM-SVR is the ones we used in the experiments in our paper.

Backup links:

Usage

For data preparation, please see directory point_sampling.

To train an autoencoder, go to IMGAN and use the following commands for progressive training. You may want to copy the commands in a .bat or .sh file.

python main.py --ae --train --epoch 50 --real_size 16 --batch_size_input 4096
python main.py --ae --train --epoch 100 --real_size 32 --batch_size_input 8192
python main.py --ae --train --epoch 200 --real_size 64 --batch_size_input 32768

The above commands will train the AE model 50 epochs in 163 resolution (each shape has 4096 sampled points), then 50 epochs in 323 resolution, and finally 100 epochs in 643 resolution.

To train a latent-gan, after training the autoencoder, use the following command to extract the latent codes:

python main.py --ae

Then train the latent-gan and get some samples:

python main.py --train --epoch 10000
python main.py

You can change some lines in main.py to adjust the number of samples and the sampling resolution.

To train the network for single-view reconstruction, after training the autoencoder, copy the weights and latent codes to the corresponding folders in IMSVR. Go to IMSVR and use the following commands to train IM-SVR and get some samples:

python main.py --train --epoch 1000
python main.py

License

This project is licensed under the terms of the MIT license (see LICENSE for details).

Comments
  • how to rot the shape in hdf5

    how to rot the shape in hdf5

    thank you for your excellent work. i have succeed in run it in my own binvox dataset.It is really pretty in Interpolation.

    but when i try to rot the 3D shape in hdf5, 1 can't solve the following problem:

    1. the hdf5 structure: data_points16 (3,4096,3) data_values16(3,4096,1) data_points32 =(3,8192,3) data_values32 =(3,8192,1) data_points64 =(3,32768,3) data_values64 =(3,32768,1) data_voxels =(3,64.64,64.1)

    why the last dim of data_points is 3 why the second dim of data_points32 is 8192,which is 161616*2

    1. rot shape method i try to rot shape by numpy.rot90, it work in data_voxels. to make sure the data set is all rot, i write the following function:
    def visualize_voxels(voxels, thres=0.5, output_dir='voxel_list', filename='out'):
        import mcubes
        """
    
        :param voxels: ndarray shape like (64,64,64,1) bool value
        :param thres: 
        :param output_dir: 
        :return: 
        """
        if not os.path.exists(output_dir):
            os.makedirs(output_dir)
    
        voxels = np.squeeze(voxels)  # (64,64,64,1)->(64,64,64)
        model_float = voxels
        vertices, triangles = mcubes.marching_cubes(model_float, thres)
        mcubes.export_mesh(vertices, triangles, os.path.join(output_dir, filename + ".dae"))
    def visualize_voxels_batch(voxels_batch, thres=0.5, output_dir='voxel_list'):
        import mcubes
        """
    
        :param voxels: ndarray shape like (64,64,64,1) bool value
        :param thres: 
        :param output_dir: 
        :return: 
        """
        if not os.path.exists(output_dir):
            os.makedirs(output_dir)
        for t in range(voxels_batch.shape[0]):
            model_float = np.squeeze(voxels_batch[t])  # (64,64,64,1)->(64,64,64)
            vertices, triangles = mcubes.marching_cubes(model_float, thres)
            mcubes.export_mesh(vertices, triangles, os.path.join(output_dir, str(t) + ".dae"))
    
    def rot_voxel2zheng(voxel_batch):
        return np.flip(np.rot90(voxel_batch,k=1,axes=(2,3)),axis=2)
    def rot_pointorvalue2zheng(points_batch):
        d=round(points_batch.shape[1]**(1/3))
        points_batch = points_batch.reshape([points_batch.shape[0],d,d,d,points_batch.shape[-1]])#(num,d**3,3)->(num,d,d,d,3)
        points_batch = np.flip(np.rot90(points_batch,k=1,axes=(2,3)),axis=2)
        return points_batch.reshape(points_batch.shape[0],d**3,points_batch.shape[-1])
    
    
    

    Because i can't understand the problem 1, I have no idea how to adjust the data_points and points and values of 32dims.

    1. other efforts in binvox after i find it hard to adjust hdf5 file. i try to adjust binvox. i use the tool in https://github.com/dimatura/binvox-rw-py but i failed in

    UnicodeEncodeError: 'gbk' codec can't encode character '\x80' in position 0: illegal multibyte sequence

    i have add my question to related issue

    opened by tianyilt 6
  • Questions about preparing data

    Questions about preparing data

    Hi, thank you very much for sharing the code of your great work. I am trying to prepare my own data for training and I use vox2pointvaluepair_from_binvox.py. I also visualize those h5py data using test_vox.py. However, I find that some shapes are good, some are not very satisfactory. image image I use binvox.exe to change the original shape from segmented data of ShapeNetV2 to voxels. I have visualized these binvox data and they look good. The shapes have been normalized to the unit box. I am wondering what causes the result in figure 2. Thanks a lot again!

    opened by FrozenSilent 4
  • how can I make gif animation from a lot of .dae file?

    how can I make gif animation from a lot of .dae file?

    Hi,sincerely thank you for your great work. Did you know how can I make gif animation from a lot of .dae file. I'm using the great work in bionic design related research and I need to use it to produce the Interpolation effect. I try many dae viewer but they can just view one model one time. one of author of the research say that he uses 3dmax in one issue and i guess it can be made by using 3dmax to make animation but i haven't taken it. I need more information about this. thank you!

    opened by tianyilt 4
  • How to test the model on my own 3D voxelized shape?

    How to test the model on my own 3D voxelized shape?

    Hi, thanks for the code first. I want to try out the model on my own 3D voxelized shape for reconstruction. Is there any quick way to do this? Only training commands are provided in README.

    Thanks!

    opened by ChrisWu1997 4
  • quantitative evaluation of 3D shape generation

    quantitative evaluation of 3D shape generation

    hi Zhiqin: Thanks for your great work. And i am doing some works about 3D shape generation recently.I find that you show the results of COV-LFD and MMD-LFD in paper.Do you have the results of COV-CD or MMD-CD ? I would appreciate it if you could reply.

    opened by flyKite1998 2
  • Why do we need hierarchical flood fill for .binvox file before sampling points?

    Why do we need hierarchical flood fill for .binvox file before sampling points?

    Hi, I'm a CV novice. Thanks for your code, it's really helpful to me. I'm trying to rewrite the code for my own research.I try to rewrite the file "vox2pointvaluepair_from_binvox.py" for sampling points from binvox of 32^3. But I can not understand the function "hierarchicalfloodFill" in the code: Why do we need to do this before sampling points. And another question: In the original function "hierarchicalfloodFill" (sampling points for binvox of 64^3), why do we need to compress voxel of 64^3 to 32^3 first and then restore it to 64^3.

    Cheers.

    opened by gaowei724 2
  • Naive questions

    Naive questions

    Dear Zhiqin,

    Thanks a lot for sharing your great work!

    We have run the script you provide successfully, but we have some naive questions as I'm not familiar with the problem ;)

    1. how should we read the output in the sample folder? Are they just rendering of the marching cube?
    2. what's the input/output in both training and testing stage exactly?
    3. how should we build our own dataset as h5 file?

    It would be great if you have time to answer these questions, thanks!

    Best, Shihao

    opened by wsh312 2
  • About resolution

    About resolution

    Hi, Zhiqin, It is a good job and I have a naive question about resolution of 3D object. Have you tried to improve the resolution of the shapes? I know that the resolution you used in paper is up to 256. The implicit field is a very good method, so can the resolution of the shape be improved without consuming a lot of memory?

    opened by duzhenjiang113 1
  • Questions on training

    Questions on training

    Hi @czq142857 , I have tried the code and its works fine.

    1.)The code in IMSVR uses the pre-trained IMAE checkpoints in IMSVR training. Can I get a more detailed explanation of using those checkpoints?

    2.) In the IMSVR\data, we have hdf5 files of train, test,only_train,only_train_z. 03001627_hsp_vox_train.hdf5 and 03001627_hsp_vox_test.hdf5 contains ['pixels', 'points_16', 'points_32', 'points_64', 'values_16', 'values_32', 'values_64']. Can you explain from how are these files made?

    Thanks!!

    opened by bharadwajdhornala 1
  • Data Preparation code

    Data Preparation code

    Thanks for great work. I wonder is it possible for you to share the code to prepare the training data? Currently, I'm having difficulty to detect if the point is inside or outside the object.

    Thanks Luan

    opened by tranluan 0
Owner
Zhiqin Chen
Video game addict.
Zhiqin Chen
This is the official source code for SLATE. We provide the code for the model, the training code, and a dataset loader for the 3D Shapes dataset. This code is implemented in Pytorch.

SLATE This is the official source code for SLATE. We provide the code for the model, the training code and a dataset loader for the 3D Shapes dataset.

Gautam Singh 66 Dec 26, 2022
Code for paper ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop.

Who Left the Dogs Out? Evaluation and demo code for our ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization

Benjamin Biggs 29 Dec 28, 2022
TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

SLM: Structural Language Models of Code This is an official implementation of the model described in: "Structural Language Models of Code" [PDF] To ap

null 73 Nov 6, 2022
Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

CoProtector Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Zhensu Sun 1 Oct 26, 2021
Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

This codebase is being actively maintained, please create and issue if you have issues using it Basics All data files are included under losses and ea

J K Terry 32 Nov 9, 2021
Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166

Region Proportion Regularized Inference (RePRI) for Few-Shot Segmentation In this repo, we provide the code for our paper : "Few-Shot Segmentation Wit

Malik Boudiaf 138 Dec 12, 2022
Code for ACM MM 2020 paper "NOH-NMS: Improving Pedestrian Detection by Nearby Objects Hallucination"

NOH-NMS: Improving Pedestrian Detection by Nearby Objects Hallucination The offical implementation for the "NOH-NMS: Improving Pedestrian Detection by

Tencent YouTu Research 64 Nov 11, 2022
Official TensorFlow code for the forthcoming paper

~ Efficient-CapsNet ~ Are you tired of over inflated and overused convolutional neural networks? You're right! It's time for CAPSULES :)

Vittorio Mazzia 203 Jan 8, 2023
This is the code for the paper "Contrastive Clustering" (AAAI 2021)

Contrastive Clustering (CC) This is the code for the paper "Contrastive Clustering" (AAAI 2021) Dependency python>=3.7 pytorch>=1.6.0 torchvision>=0.8

Yunfan Li 210 Dec 30, 2022
Code for the paper Learning the Predictability of the Future

Learning the Predictability of the Future Code from the paper Learning the Predictability of the Future. Website of the project in hyperfuture.cs.colu

Computer Vision Lab at Columbia University 139 Nov 18, 2022
PyTorch code for the paper: FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning

FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning This is the PyTorch implementation of our paper: FeatMatch: Feature-Based Augmentat

null 43 Nov 19, 2022
Code for the paper A Theoretical Analysis of the Repetition Problem in Text Generation

A Theoretical Analysis of the Repetition Problem in Text Generation This repository share the code for the paper "A Theoretical Analysis of the Repeti

Zihao Fu 37 Nov 21, 2022
Code for our ICASSP 2021 paper: SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

SA-Net: Shuffle Attention for Deep Convolutional Neural Networks (paper) By Qing-Long Zhang and Yu-Bin Yang [State Key Laboratory for Novel Software T

Qing-Long Zhang 199 Jan 8, 2023
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Facebook Research 296 Dec 29, 2022
Code for the Shortformer model, from the paper by Ofir Press, Noah A. Smith and Mike Lewis.

Shortformer This repository contains the code and the final checkpoint of the Shortformer model. This file explains how to run our experiments on the

Ofir Press 138 Apr 15, 2022
PyTorch code for ICLR 2021 paper Unbiased Teacher for Semi-Supervised Object Detection

Unbiased Teacher for Semi-Supervised Object Detection This is the PyTorch implementation of our paper: Unbiased Teacher for Semi-Supervised Object Detection

Facebook Research 366 Dec 28, 2022
Official code for paper "Optimization for Oriented Object Detection via Representation Invariance Loss".

Optimization for Oriented Object Detection via Representation Invariance Loss By Qi Ming, Zhiqiang Zhou, Lingjuan Miao, Xue Yang, and Yunpeng Dong. Th

ming71 56 Nov 28, 2022
Code for our CVPR 2021 paper "MetaCam+DSCE"

Joint Noise-Tolerant Learning and Meta Camera Shift Adaptation for Unsupervised Person Re-Identification (CVPR'21) Introduction Code for our CVPR 2021

FlyingRoastDuck 59 Oct 31, 2022
Code for our CVPR2021 paper coordinate attention

Coordinate Attention for Efficient Mobile Network Design (preprint) This repository is a PyTorch implementation of our coordinate attention (will appe

Qibin (Andrew) Hou 726 Jan 5, 2023