ICCV2021 Oral SA-ConvONet: Sign-Agnostic Optimization of Convolutional Occupancy Networks

Overview

Sign-Agnostic Convolutional Occupancy Networks

Paper | Supplementary | Video | Teaser Video | Project Page

This repository contains the implementation of the paper:

Sign-Agnostic CONet: Learning Implicit Surface Reconstructions by Sign-Agnostic Optimization of Convolutional Occupancy Networks ICCV 2021 (Oral)

If you find our code or paper useful, please consider citing

@inproceedings{tang2021sign,
  title={SA-ConvONet: Sign-Agnostic Optimization of Convolutional Occupancy Networks},
  author={Tang, Jiapeng and Lei, Jiabao and Xu, Dan and Ma, Feiying and Jia, Kui and Zhang, Lei},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  year={2021}
}

Contact Jiapeng Tang for questions, comments and reporting bugs.

Installation

First you have to make sure that you have all dependencies in place. The simplest way to do so, is to use anaconda.

You can create an anaconda environment called sa_conet using

conda env create -f environment.yaml
conda activate sa_conet

Note: you might need to install torch-scatter mannually following the official instruction:

pip install torch-scatter==2.0.4 -f https://pytorch-geometric.com/whl/torch-1.4.0+cu101.html

Next, compile the extension modules. You can do this via

python setup.py build_ext --inplace

Demo

First, run the script to get the demo data:

bash scripts/download_demo_data.sh

Reconstruct Large-Scale Matterport3D Scene

You can now quickly test our code on the real-world scene shown in the teaser. To this end, simply run:

python generate_optim_largescene.py configs/pointcloud_crop/demo_matterport.yaml

This script should create a folder out/demo_matterport/generation where the output meshes and input point cloud are stored.

Note: This experiment corresponds to our fully convolutional model, which we train only on the small crops from our synthetic room dataset. This model can be directly applied to large-scale real-world scenes with real units and generate meshes in a sliding-window manner, as shown in the teaser. More details can be found in section D.1 of our supplementary material. For training, you can use the script pointcloud_crop/room_grid64.yaml.

Reconstruct Synthetic Indoor Scene

You can also test on our synthetic room dataset by running:

python generate_optim_scene.py configs/pointcloud/demo_syn_room.yaml

Reconstruct ShapeNet Object

You can also test on the ShapeNet dataset by running:

python generate_optim_object.py configs/pointcloud/demo_shapenet.yaml --this file needs to be created.

Dataset

To evaluate a pretrained model or train a new model from scratch, you have to obtain the respective dataset. In this paper, we consider 4 different datasets:

ShapeNet

You can download the dataset (73.4 GB) by running the script from Occupancy Networks. After, you should have the dataset in data/ShapeNet folder.

Synthetic Indoor Scene Dataset

For scene-level reconstruction, we use a synthetic dataset of 5000 scenes with multiple objects from ShapeNet (chair, sofa, lamp, cabinet, table). There are also ground planes and randomly sampled walls.

You can download the preprocessed data (144 GB) by ConvONet using

bash scripts/download_data.sh

This script should download and unpack the data automatically into the data/synthetic_room_dataset folder.
Note: The point-wise semantic labels are also provided in the dataset, which might be useful.

Alternatively, you can also preprocess the dataset yourself. To this end, you can:

  • download the ShapeNet dataset as described above.
  • check scripts/dataset_synthetic_room/build_dataset.py, modify the path and run the code.

Matterport3D

Download Matterport3D dataset from the official website. And then, use scripts/dataset_matterport/build_dataset.py to preprocess one of your favorite scenes. Put the processed data into data/Matterport3D_processed folder.

ScanNet

Download ScanNet v2 data from the official ScanNet website. Then, you can preprocess data with: scripts/dataset_scannet/build_dataset.py and put into data/ScanNet folder.
Note: Currently, the preprocess script normalizes ScanNet data to a unit cube for the comparison shown in the paper, but you can easily adapt the code to produce data with real-world metric. You can then use our fully convolutional model to run evaluation in a sliding-window manner.

Usage

When you have installed all binary dependencies and obtained the preprocessed data, you are ready to perform sign-agnostic optimzation, run the pre-trained models, and train new models from scratch.

Mesh Generation for ConvOnet

To generate meshes using a pre-trained model, use

python generate.py CONFIG.yaml

where you replace CONFIG.yaml with the correct config file.

Use pre-trained models The easiest way is to use a pre-trained model. You can do this by using one of the config files under the pretrained folders.

For example, for 3D reconstruction from noisy point cloud with our 3-plane model on the synthetic room dataset, you can simply run:

python generate.py configs/pointcloud/pretrained/room_3plane.yaml

The script will automatically download the pretrained model and run the mesh generation. You can find the outputs in the out/.../generation_pretrained folders

Note that the config files are only for generation, not for training new models: when these configs are used for training, the model will be trained from scratch, but during inference our code will still use the pretrained model.

The provided following pretrained models are:

pointcloud/shapenet_3plane.pt
pointcloud/room_grid64.pt
pointcloud_crop/room_grid64.pt

Sign-Agnostic Optimization of ConvONet

Before the sign-agnostic, test-time optimization on the Matterport3D dataset, we firstly run the below script to preprocess the testset.

python scripts/dataset_matterport/make_cropscene_dataset.py --in_folder $in_folder --out_folder $out_folder --do_norm

Please specify the in_folder and out_folder.

To perform sign-agnostic, test-time optimization for more accurate surface mesh generation using a pretrained model, use

python generate_optim_object.py configs/pointcloud/test_optim/shapenet_3plane.yaml
python generate_optim_scene.py configs/pointcloud/test_optim/room_grid64.yaml
python generate_optim_largescene.py configs/pointcloud_crop/test_optim/room_grid64.yaml

Evaluation

For evaluation of the models, we provide the scripts eval_meshes.py and eval_meshes_optim.py. You can run it using:

python eval_meshes.py CONFIG.yaml
python eval_meshes_optim.py CONFIG.yaml

The scripts takes the meshes generated in the previous step and evaluates them using a standardized protocol. The output will be written to .pkl/.csv files in the corresponding generation folder which can be processed using pandas.

Training

Finally, to pretrain a new network from scratch, run:

python train.py CONFIG.yaml

For available training options, please take a look at configs/default.yaml.

Acknowledgements

Most of the code is borrowed from ConvONet. We thank Songyou Peng for his great works.

Comments
  • Which data is used under the synthetic room dataset

    Which data is used under the synthetic room dataset

    Hi, thanks for your work.

    I have noticed that you randomly picked 50 shapes for each set (train/val/test) under the synthetic room dataset.

    Would you like to share which data you have used (e.g., index)? I want to do the same exp as you.

    Looking forward to your reply.

    opened by bearprin 2
  • Is there any method to preserve the original color of the point cloud on the output model?

    Is there any method to preserve the original color of the point cloud on the output model?

    Hi, @tangjiapeng I'm working on a study project to reconstruct the outdoor environment via point cloud data and trying to use your impressive method on it. I just found out that the model output .off file, which is not supported to save the color. Since I want to make an end-to-end pipeline, I'm wondering if there is a way to preserve the original color of the point cloud on the output model?

    opened by TKPTKPTKP 1
  • How can I call multiple GPUs during training

    How can I call multiple GPUs during training

    Hello @tangjiapeng Your work is great and i am very interested in your article, and I have some questions to ask you. I have five GPUs, how to call multiple GPUs to run code during training?

    opened by miaowang525 1
  • GPU memory issue when running the demo

    GPU memory issue when running the demo

    Hi,

    I am facing a memory issue where CUDA complains about insufficient VRAM. I noticed this issue stems from line 245 of the file src/conv_onet/training.py where it tries to extract 3D features from the input point cloud.

    My GPU has 8GiB VRAM but it seems the demo needs more than that. Can you tell me what's the minimum required VRAM size and perhaps there is a way to reduce the VRAM requirement? Thanks

    Regards

    opened by xingruiy 4
  • Error running `generate_optim_largescene.py configs/pointcloud_crop/demo_matterport.yaml`

    Error running `generate_optim_largescene.py configs/pointcloud_crop/demo_matterport.yaml`

    I've downloaded the dataset using bash scripts/download_demo_data.sh

    (sa_conet) satyajit@cloud-pc:~/project/SA-ConvONet$ python3 generate_optim_largescene.py configs/pointcloud_crop/demo_matterport.yaml
    /home/satyajit/project/SA-ConvONet/src/config.py:22: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
      cfg_special = yaml.load(f)
    /home/satyajit/project/SA-ConvONet/src/config.py:33: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
      cfg = yaml.load(f)
    test_split: test
    https://s3.eu-central-1.amazonaws.com/avg-projects/convolutional_occupancy_networks/models/pointcloud_crop/room_grid64.pt
    => Loading checkpoint from url...
    Warning: generator does not support pointcloud generation.
      0%|                                                                                                                   | 0/2 [00:00<?, ?it/s]Error occured when loading field points of model JmbYfDe2QKZ
      0%|                                                                                                                   | 0/2 [00:00<?, ?it/s]
    Traceback (most recent call last):
      File "generate_optim_largescene.py", line 78, in <module>
        for it, data in enumerate(tqdm(test_loader)):
      File "/home/satyajit/.local/lib/python3.6/site-packages/tqdm/std.py", line 1171, in __iter__
        for obj in iterable:
      File "/home/satyajit/anaconda3/envs/sa_conet/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
        data = self._next_data()
      File "/home/satyajit/anaconda3/envs/sa_conet/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
        data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
      File "/home/satyajit/anaconda3/envs/sa_conet/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
        return self.collate_fn(data)
      File "/home/satyajit/anaconda3/envs/sa_conet/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 81, in default_collate
        raise TypeError(default_collate_err_msg_format.format(elem_type))
    TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'NoneType'>
    
    opened by satyajitghana 2
Owner
null
PyTorch implementation for Stochastic Fine-grained Labeling of Multi-state Sign Glosses for Continuous Sign Language Recognition.

Stochastic CSLR This is the PyTorch implementation for the ECCV 2020 paper: Stochastic Fine-grained Labeling of Multi-state Sign Glosses for Continuou

Zhe Niu 28 Dec 19, 2022
Predicting Semantic Map Representations from Images with Pyramid Occupancy Networks

This is the code associated with the paper Predicting Semantic Map Representations from Images with Pyramid Occupancy Networks, published at CVPR 2020.

Thomas Roddick 219 Dec 20, 2022
This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of Coordinate Independent Convolutional Networks.

Orientation independent Möbius CNNs This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of

Maurice Weiler 59 Dec 9, 2022
Implementation of ICCV2021(Oral) paper - VMNet: Voxel-Mesh Network for Geodesic-aware 3D Semantic Segmentation

VMNet: Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation Created by Zeyu HU Introduction This work is based on our paper VMNet: Voxel-Mes

HU Zeyu 82 Dec 27, 2022
TOOD: Task-aligned One-stage Object Detection, ICCV2021 Oral

One-stage object detection is commonly implemented by optimizing two sub-tasks: object classification and localization, using heads with two parallel branches, which might lead to a certain level of spatial misalignment in predictions between the two tasks.

null 264 Jan 9, 2023
Official implementation of "A Unified Objective for Novel Class Discovery", ICCV2021 (Oral)

A Unified Objective for Novel Class Discovery This is the official repository for the paper: A Unified Objective for Novel Class Discovery Enrico Fini

Enrico Fini 118 Dec 26, 2022
Official PyTorch Implementation of Convolutional Hough Matching Networks, CVPR 2021 (oral)

Convolutional Hough Matching Networks This is the implementation of the paper "Convolutional Hough Matching Network" by J. Min and M. Cho. Implemented

Juhong Min 70 Nov 22, 2022
Focal Sparse Convolutional Networks for 3D Object Detection (CVPR 2022, Oral)

Focal Sparse Convolutional Networks for 3D Object Detection (CVPR 2022, Oral) This is the official implementation of Focals Conv (CVPR 2022), a new sp

DV Lab 280 Jan 7, 2023
Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

AngularGrad Optimizer This repository contains the oficial implementation for AngularGrad: A New Optimization Technique for Angular Convergence of Con

mario 124 Sep 16, 2022
Repository for the paper "Online Domain Adaptation for Occupancy Mapping", RSS 2020

RSS 2020 - Online Domain Adaptation for Occupancy Mapping Repository for the paper "Online Domain Adaptation for Occupancy Mapping", Robotics: Science

Anthony 26 Sep 22, 2022
PyTorch implementation of SMODICE: Versatile Offline Imitation Learning via State Occupancy Matching

SMODICE: Versatile Offline Imitation Learning via State Occupancy Matching This is the official PyTorch implementation of SMODICE: Versatile Offline I

Jason Ma 14 Aug 30, 2022
Nest Protect integration for Home Assistant. This will allow you to integrate your smoke, heat, co and occupancy status real-time in HA.

Nest Protect integration for Home Assistant Custom component for Home Assistant to interact with Nest Protect devices via an undocumented and unoffici

Mick Vleeshouwer 175 Dec 29, 2022
[CVPR'22] COAP: Learning Compositional Occupancy of People

COAP: Compositional Articulated Occupancy of People Paper | Video | Project Page This is the official implementation of the CVPR 2022 paper COAP: Lear

Marko Mihajlovic 111 Dec 11, 2022
Project for tracking occupancy in Tel-Aviv parking lots.

Ahuzat Dibuk - Tracking occupancy in Tel-Aviv parking lots main.py This module was set-up to be executed on Google Cloud Platform. I run it every 15 m

Geva Kipper 35 Nov 22, 2022
This project demonstrates the use of neural networks and computer vision to create a classifier that interprets the Brazilian Sign Language.

LIBRAS-Image-Classifier This project demonstrates the use of neural networks and computer vision to create a classifier that interprets the Brazilian

Aryclenio Xavier Barros 26 Oct 14, 2022
Official code for "End-to-End Optimization of Scene Layout" -- including VAE, Diff Render, SPADE for colorization (CVPR 2020 Oral)

End-to-End Optimization of Scene Layout Code release for: End-to-End Optimization of Scene Layout CVPR 2020 (Oral) Project site, Bibtex For help conta

Andrew Luo 41 Dec 9, 2022
[ICCV 2021 Oral] NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

NerfingMVS Project Page | Paper | Video | Data NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo Yi Wei, Shaohui

Yi Wei 369 Dec 24, 2022
Official PyTorch code of DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization (ICCV 2021 Oral).

DeepPanoContext (DPC) [Project Page (with interactive results)][Paper] DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context G

Cheng Zhang 66 Nov 16, 2022
Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Optimization Algorithm,Immune Algorithm, Artificial Fish Swarm Algorithm, Differential Evolution and TSP(Traveling salesman)

scikit-opt Swarm Intelligence in Python (Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Algorithm, Immune Algorithm,A

郭飞 3.7k Jan 3, 2023