PyTorch 1.5 implementation for paper DECOR-GAN: 3D Shape Detailization by Conditional Refinement.

Overview

DECOR-GAN

PyTorch 1.5 implementation for paper DECOR-GAN: 3D Shape Detailization by Conditional Refinement, Zhiqin Chen, Vladimir G. Kim, Matthew Fisher, Noam Aigerman, Hao Zhang, Siddhartha Chaudhuri.

Paper | Oral video | GUI demo video

Citation

If you find our work useful in your research, please consider citing:

@article{chen2021decor,
  title={DECOR-GAN: 3D Shape Detailization by Conditional Refinement},
  author={Zhiqin Chen and Vladimir G. Kim and Matthew Fisher and Noam Aigerman and Hao Zhang and Siddhartha Chaudhuri},
  journal={Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021}
}

Dependencies

Requirements:

  • Python 3.6 with numpy, h5py, scipy, sklearn and Cython
  • PyTorch 1.5 (other versions may also work)
  • PyMCubes (for marching cubes)
  • OpenCV-Python (for reading and writing images)

Build Cython module:

python setup.py build_ext --inplace

Datasets and pre-trained weights

For data preparation, please see data_preparation.

We provide the ready-to-use datasets here.

Backup links:

We also provide the pre-trained network weights.

Backup links:

Training

To train the network:

python main.py --data_style style_chair_64 --data_content content_chair_train --data_dir ./data/03001627/ --alpha 0.5 --beta 10.0 --input_size 32 --output_size 128 --train --gpu 0 --epoch 20
python main.py --data_style style_plane_32 --data_content content_plane_train --data_dir ./data/02691156/ --alpha 0.1 --beta 10.0 --input_size 64 --output_size 256 --train --gpu 0 --epoch 20
python main.py --data_style style_car_32 --data_content content_car_train --data_dir ./data/02958343/ --alpha 0.2 --beta 10.0 --input_size 64 --output_size 256 --train --gpu 0 --epoch 20
python main.py --data_style style_table_64 --data_content content_table_train --data_dir ./data/04379243/ --alpha 0.2 --beta 10.0 --input_size 16 --output_size 128 --train --gpu 0 --epoch 50
python main.py --data_style style_motor_16 --data_content content_motor_all_repeat20 --data_dir ./data/03790512/ --alpha 0.5 --beta 10.0 --input_size 64 --output_size 256 --train --asymmetry --gpu 0 --epoch 20
python main.py --data_style style_laptop_32 --data_content content_laptop_all_repeat5 --data_dir ./data/03642806/ --alpha 0.2 --beta 10.0 --input_size 32 --output_size 256 --train --asymmetry --gpu 0 --epoch 20
python main.py --data_style style_plant_20 --data_content content_plant_all_repeat8 --data_dir ./data/03593526_03991062/ --alpha 0.5 --beta 10.0 --input_size 32 --output_size 256 --train --asymmetry --gpu 0 --epoch 20

Note that style_chair_64 means the model will be trained with 64 detailed chairs. You can modify the list of detailed shapes in folder splits, such as style_chair_64.txt. You can also modify the list of content shapes in folder splits. The parameters input_size and output_size specify the resolutions of the input and output voxels. Valid settings are as follows:

Input resolution Output resolution Upsampling rate
64 256 x4
32 128 x4
32 256 x8
16 128 x8

GUI application

To launch UI for a pre-trained model, replace --data_content to the testing content shapes and replace --train with --ui.

python main.py --data_style style_chair_64 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 128 --ui --gpu 0

Testing

These are examples for testing a model trained with 32 detailed chairs. For others, please change the commands accordingly.

Rough qualitative testing

To output a few detailization results (the first 16 content shapes x 32 styles) and a T-SNE embedding of the latent space:

python main.py --data_style style_chair_32 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 128 --test --gpu 0

The output images can be found in folder samples.

IOU, LP, Div

To test Strict-IOU, Loose-IOU, LP-IOU, Div-IOU, LP-F-score, Div-F-score:

python main.py --data_style style_chair_64 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 128 --prepvoxstyle --gpu 0
python main.py --data_style style_chair_32 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 128 --prepvox --gpu 0
python main.py --data_style style_chair_64 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 128 --evalvox --gpu 0

The first command prepares the patches in 64 detailed training shapes, thus --data_style is style_chair_64. Specifically, it removes duplicated patches in each detailed training shape and only keep unique patches for faster computation in the following testing procedure. The unique patches are written to folder unique_patches. Note that if you are testing multiple models, you do not have to run the first command every time -- just copy the folder unique_patches or make a symbolic link.

The second command runs the model and outputs the detailization results, in folder output_for_eval.

The third command evaluates the outputs. The results are written to folder eval_output ( result_IOU_mean.txt, result_LP_Div_Fscore_mean.txt, result_LP_Div_IOU_mean.txt ).

Cls-score

To test Cls-score:

python main.py --data_style style_chair_64 --data_content content_chair_all --data_dir ./data/03001627/ --input_size 32 --output_size 128 --prepimgreal --gpu 0
python main.py --data_style style_chair_32 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 128 --prepimg --gpu 0
python main.py --data_style style_chair_64 --data_content content_chair_all --data_dir ./data/03001627/ --input_size 32 --output_size 128 --evalimg --gpu 0

The first command prepares rendered views of all content shapes, thus --data_content is content_chair_all. The rendered views are written to folder render_real_for_eval. Note that if you are testing multiple models, you do not have to run the first command every time -- just copy the folder render_real_for_eval or make a symbolic link.

The second command runs the model and outputs rendered views of the detailization results, in folder render_fake_for_eval.

The third command evaluates the outputs. The results are written to folder eval_output ( result_Cls_score.txt ).

FID

To test FID-all and FID-style, you need to first train a classification model on shapeNet. You can use the provided pre-trained weights here (Clsshapenet_128.pth and Clsshapenet_256.pth for 1283 and 2563 inputs).

Backup links:

In case you need to train your own model, modify shapenet_dir in evalFID.py and run:

python main.py --prepFIDmodel --output_size 128 --gpu 0
python main.py --prepFIDmodel --output_size 256 --gpu 0

After you have the pre-trained classifier, use the following commands:

python main.py --data_style style_chair_64 --data_content content_chair_all --data_dir ./data/03001627/ --input_size 32 --output_size 128 --prepFIDreal --gpu 0
python main.py --data_style style_chair_32 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 128 --prepFID --gpu 0
python main.py --data_style style_chair_64 --data_content content_chair_all --data_dir ./data/03001627/ --input_size 32 --output_size 128 --evalFID --gpu 0

The first command computes the mean and sigma vectors for real shapes and writes to precomputed_real_mu_sigma_128_content_chair_all_num_style_16.hdf5. Note that if you are testing multiple models, you do not have to run the first command every time -- just copy the output hdf5 file or make a symbolic link.

The second command runs the model and outputs the detailization results, in folder output_for_FID.

The third command evaluates the outputs. The results are written to folder eval_output ( result_FID.txt ).

You might also like...
Unoffical implementation about Image Super-Resolution via Iterative Refinement by Pytorch
Unoffical implementation about Image Super-Resolution via Iterative Refinement by Pytorch

Image Super-Resolution via Iterative Refinement Paper | Project Brief This is a unoffical implementation about Image Super-Resolution via Iterative Re

FuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space OptimizationFuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space Optimization
FuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space OptimizationFuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space Optimization

FuseDream This repo contains code for our paper (paper link): FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizat

Pytorch implementation of the paper "Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization"

Pytorch implementation of the paper "Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization"

This repo is a PyTorch implementation for Paper
This repo is a PyTorch implementation for Paper "Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds"

Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds This repository is a PyTorch implementation for paper: Uns

PyTorch implementation of
PyTorch implementation of "Representing Shape Collections with Alignment-Aware Linear Models" paper.

deep-linear-shapes PyTorch implementation of "Representing Shape Collections with Alignment-Aware Linear Models" paper. If you find this code useful i

Official PyTorch code for WACV 2022 paper "CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows"

CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows WACV 2022 preprint:https://arxiv.org/abs/2107.1

Official implementation for (Refine Myself by Teaching Myself : Feature Refinement via Self-Knowledge Distillation, CVPR-2021)

FRSKD Official implementation for Refine Myself by Teaching Myself : Feature Refinement via Self-Knowledge Distillation (CVPR-2021) Requirements Pytho

Official Implementation for
Official Implementation for "ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement" https://arxiv.org/abs/2104.02699

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement Recently, the power of unconditional image synthesis has significantly advanced th

Paddle implementation for "Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation" (NAACL 2021)

L1-Refinement Paddle implementation for "Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation" (NAACL 2021) 🙈 A more detailed readme is co

Comments
  • understanding data preparation

    understanding data preparation

    Hello, thanks for publishing your code :)

    I have the following questions regarding the 3 possible scripts for step 3:

    • is there some documentation on how depth fusion is done & what is the difference between depth fusion and flood fill approach?
    • Is the dimension 2560 for rendering variable arbitrarily chosen? and why is state_ctr shape 512x512x64, 2?
    • what should be the difference between the initial .binvox created in step 2 and the .binvox created in step 3?

    Thanks in advance

    opened by christinazavou 2
  • Suggesting bug fix in get_voxel_bbox

    Suggesting bug fix in get_voxel_bbox

    Hello, I'd like to suggest a bug fix I found while running the code on my meshes ... I don't know if it was on purpose, but the current implementation of get_voxel_bbox is always returning 1 as xmin, ymin, zmin, and never 0, thus my voxelized meshes were cut when crop_voxel method was called.

    Therefore I modified get_voxel_bbox into the following:

        def get_voxel_bbox(self, vox):
            # minimap
            vox_tensor = torch.from_numpy(vox).to(self.device).unsqueeze(0).unsqueeze(0).float()
            smallmaskx_tensor = F.max_pool3d(vox_tensor, kernel_size=self.upsample_rate, stride=self.upsample_rate,
                                             padding=0)
            smallmaskx = smallmaskx_tensor.detach().cpu().numpy()[0, 0]
            smallmaskx = np.round(smallmaskx).astype(np.uint8)
            # x
            ray = np.max(smallmaskx, (1, 2))
            indices = np.where(ray == 1)
            xmin = indices[0][0]
            xmax = indices[0][-1]
            # y
            ray = np.max(smallmaskx, (0, 2))
            indices = np.where(ray == 1)
            ymin = indices[0][0]
            ymax = indices[0][-1]
            # z
            ray = np.max(smallmaskx, (0, 1))
            if self.asymmetry:
                indices = np.where(ray == 1)
                zmin = indices[0][0]
                zmax = indices[0][-1]
            else:
                raise Exception("cant")
            return xmin, xmax + 1, ymin, ymax + 1, zmin, zmax + 1
    

    Below are examples of the cropped voxelized mesh with the old implementation and the new implementation.

    image

    image

    image

    image

    image

    image

    image

    image

    For my visualizations I used the following code:

    def get_points_from_voxel(vox_model):
        xp, yp, zp = np.where(vox_model == 1)
        normalized_mid_point_xs = np.linspace(0, 1, vox_model.shape[0]) + 1 / (vox_model.shape[0] * 2)
        normalized_mid_point_ys = np.linspace(0, 1, vox_model.shape[1]) + 1 / (vox_model.shape[1] * 2)
        normalized_mid_point_zs = np.linspace(0, 1, vox_model.shape[2]) + 1 / (vox_model.shape[2] * 2)
        xp = normalized_mid_point_xs[xp]
        yp = normalized_mid_point_ys[yp]
        zp = normalized_mid_point_zs[zp]
        points = np.vstack([xp, yp, zp]).T
        return points
    
    
    def get_bound_points_from_voxel(vox_model):
        xp = np.array([0,0,0,0,1,1,1,1]) * (vox_model.shape[0]-1)
        yp = np.array([0,0,1,1,0,0,1,1]) * (vox_model.shape[1]-1)
        zp = np.array([0,1,0,1,0,1,0,1]) * (vox_model.shape[2]-1)
        normalized_mid_point_xs = np.linspace(0, 1, vox_model.shape[0]) + 1 / (vox_model.shape[0] * 2)
        normalized_mid_point_ys = np.linspace(0, 1, vox_model.shape[1]) + 1 / (vox_model.shape[1] * 2)
        normalized_mid_point_zs = np.linspace(0, 1, vox_model.shape[2]) + 1 / (vox_model.shape[2] * 2)
        xp = normalized_mid_point_xs[xp]
        yp = normalized_mid_point_ys[yp]
        zp = normalized_mid_point_zs[zp]
        points = np.vstack([xp, yp, zp]).T
        return points
    
    
    def PointCloud(points, colors=None):
        pcd = o3d.geometry.PointCloud()
        pcd.points = o3d.utility.Vector3dVector(points)
        if colors is not None:
            pcd.colors = o3d.utility.Vector3dVector(colors)
        return pcd
    
    
    def interactive_plot(meshes):
        viewer = open3d.visualization.VisualizerWithKeyCallback()
        viewer.create_window()
        for mesh in meshes:
            viewer.add_geometry(mesh)
        viewer.run()
        viewer.destroy_window()
    
    
    tmp_raw = get_vox_from_binvox_1over2(binvox_file).astype(np.uint8)
    xmin1, xmax1, ymin1, ymax1, zmin1, zmax1 = dset.get_voxel_bbox(tmp_raw)
    tmp_raw_cropped1 = crop_voxel(tmp_raw, xmin1, xmax1, ymin1, ymax1, zmin1, zmax1)
    
    points_cropped1vox = get_points_from_voxel(tmp_raw_cropped1)
    bound_points_cropped1vox = get_bound_points_from_voxel(tmp_raw_cropped1)
    reds = np.array([[255,0,0]]*len(bound_points_pooled_tmp_raw1))
    blues = np.array([[0,0,255]]*len(points_pooled_tmp_raw1))
    interactive_plot([PointCloud(bound_points_cropped1vox, reds), PointCloud(points_cropped1vox, blues)])
    
    
    opened by christinazavou 1
Owner
Zhiqin Chen
Video game addict.
Zhiqin Chen
Official implementation of the paper Chunked Autoregressive GAN for Conditional Waveform Synthesis

Chunked Autoregressive GAN (CARGAN) Official implementation of the paper Chunked Autoregressive GAN for Conditional Waveform Synthesis [paper] [compan

Descript 150 Dec 6, 2022
PyTorch implementation for OCT-GAN Neural ODE-based Conditional Tabular GANs (WWW 2021)

OCT-GAN: Neural ODE-based Conditional Tabular GANs (OCT-GAN) Code for reproducing the experiments in the paper: Jayoung Kim*, Jinsung Jeon*, Jaehoon L

BigDyL 7 Dec 27, 2022
House-GAN++: Generative Adversarial Layout Refinement Network towards Intelligent Computational Agent for Professional Architects

House-GAN++ Code and instructions for our paper: House-GAN++: Generative Adversarial Layout Refinement Network towards Intelligent Computational Agent

null 122 Dec 28, 2022
Official pytorch code for SSC-GAN: Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation(ICCV 2021)

SSC-GAN_repo Pytorch implementation for 'Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation'.PDF SSC-GAN:Sem

tyty 4 Aug 28, 2022
[CVPR 2021] Unsupervised 3D Shape Completion through GAN Inversion

ShapeInversion Paper Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, Chen Change Loy "Unsupervised 3D

null 100 Dec 22, 2022
Implementation of the paper "Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning"

Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning This is the implementation of the paper "Self-Promoted Prototype Refinement

Kai Zhu 78 Dec 2, 2022
PyTorch Implementation of Google Brain's WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis

WaveGrad2 - PyTorch Implementation PyTorch Implementation of Google Brain's WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis. Status (202

Keon Lee 59 Dec 6, 2022
Pytorch implementation of “Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement”

Graph-to-Graph Transformers Self-attention models, such as Transformer, have been hugely successful in a wide range of natural language processing (NL

Idiap Research Institute 40 Aug 14, 2022
Implementation of Invariant Point Attention, used for coordinate refinement in the structure module of Alphafold2, as a standalone Pytorch module

Invariant Point Attention - Pytorch Implementation of Invariant Point Attention as a standalone module, which was used in the structure module of Alph

Phil Wang 113 Jan 5, 2023
Photographic Image Synthesis with Cascaded Refinement Networks - Pytorch Implementation

Photographic Image Synthesis with Cascaded Refinement Networks-Pytorch (https://arxiv.org/abs/1707.09405) This is a Pytorch implementation of cascaded

Soumya Tripathy 63 Mar 27, 2022