SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images

Overview

SymmetryNet

SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images

ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2020)

Created by Yifei Shi, Junwen Huang, Hongjia Zhang, Xin Xu, Szymon Rusinkiewicz and Kai Xu

teaser

This repository includes:

  • tools: the training scripts and evaluation scripts
    • tools/train_shapenet.py: the training script for shapenet dataset
    • tools/train_ycb.py: the training script for ycb dataset
    • tools/train_scannet.py: the training script for scannet dataset
    • tools/evaluation: the evaluation scripts
      • evaluation/eval_ref_shapenet.py: the evaluation script for reflectional symmetry on shapenet dataset
      • evaluation/eval_ref_ycb.py: the evaluation script for reflectional symmetry on ycb dataset
      • evaluation/eval_ref_scannet.py: the evaluation script for reflectional symmetry on scannet dataset
      • evaluation/eval_rot_shapenet.py: the evaluation script for rotational symmetry on shapenet dataset
      • evaluation/eval_rot_ycb.py: the evaluation script for rotational symmetry on ycb dataset
      • evaluation/eval_rot_scannet.py: the evaluation script for rotational symmetry on scannet dataset
  • lib: the core Python library for networks and loss
    • lib/loss.py: symmetrynet loss caculation for both reflectional and rotational symmetries,the loss items are listed at the end of the text
    • lib/network.py: network architecture
    • lib/tools.py: functions for the operation of rotation and reflection
    • lib/verification.py: verification of the rotational and reflectional symmetries
  • datasets: the dataloader and training/testing lists
    • datasets/shapenet/dataset.py: the training dataloader for shapnet dataset
    • datasets/shapenet/dataset_eval.py: the evaluation dataloader for shapnet dataset
      • datasets/shapenet/dataset_config/*.txt: training and testing splits for shapenet dataset, the testing splits includ holdout view/instance/category
    • datasets/ycb/dataset.py: the training dataloader for ycb dataset
    • datasets/ycb/dataset_eval.py: the evaluation dataloader for ycb dataset
      • datasets/ycb/dataset_config/*.txt: training and testing splits for shapenet dataset,the training/testing splits fallow the ycb defult settings
    • datasets/shapenet/dataset.py: the training dataloader for scannet dataset
    • datasets/shapenet/dataset_eval.py: the evaluation dataloader for scannet dataset
      • datasets/scannet/dataset_config/*.txt: training and testing splits for scannet dataset,the testing splits includ holdout view/scene

Environments

pytorch>=0.4.1 python >=3.6

Datasets

  • ShapeNet dataset

    • shapenetcore: this folder saves the models and their ground truth symmetries for each instance
    • rendered_data: this folder saves the rgbd images that we rendered for each instance, including their ground truth pose and camera intrinsic matrix, etc.
    • name_list.txt: this file saves the correspondence between the name of instances and their ID in this project(the names are too long to identify)
  • YCB dataset

    • models: this folder saves the ground truth model symmetry for each instance
    • data: this folder saves the rgbd videos and the ground truth poses and camera information
    • classes.txt: this file saves the correspondence between the name of YCB objects and their *.xyz models
    • symmetries.txt: this file saves all the ground truth symmetries for ycb object models

Training

To train the network with the default parameter on shapenet dataset, run

python tools/train_shapenet.py --dataset_root= your/folder/to/shapnet/dataset

To train the network with the default parameter on ycb dataset, run

python tools/train_ycb.py --dataset_root= your/folder/to/ycb/dataset

To train the network with the default parameter on scannet dataset, run

python tools/train_scannet.py --dataset_root= your/folder/to/scannet/dataset

Evaluation

To evaluate the model with our metric on shapenet, for reflectional symmetry, run

python tools/evaluation/eval_ref_shapenet.py

for rotational symmetry, run

python tools/evaluation/eval_rot_shapenet.py

To evaluate the model with our metric on ycb, for reflectional symmetry, run

python tools/evaluation/eval_ref_ycb.py

for rotational symmetry, run

python tools/evaluation/eval_rot_ycb.py

To evaluate the model with our metric on scannet, for reflectional symmetry, run

python tools/evaluation/eval_ref_scannet.py

for rotational symmetry, run

python tools/evaluation/eval_rot_scannet.py

Pretrained model & data download

The pretrained models and data can be found at here (dropbox) and here (baidu yunpan, password: symm).

You might also like...
CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image.
CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image.

CoReNet CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image. It produces coherent reconstructions, where all objec

Official PyTorch implementation of
Official PyTorch implementation of "Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image", ICCV 2019

PoseNet of "Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image" Introduction This repo is official Py

Single-stage Keypoint-based Category-level Object Pose Estimation from an RGB Image
Single-stage Keypoint-based Category-level Object Pose Estimation from an RGB Image

CenterPose Overview This repository is the official implementation of the paper "Single-stage Keypoint-based Category-level Object Pose Estimation fro

 Learning Skeletal Articulations with Neural Blend Shapes
Learning Skeletal Articulations with Neural Blend Shapes

This repository provides an end-to-end library for automatic character rigging and blend shapes generation as well as a visualization tool. It is based on our work Learning Skeletal Articulations with Neural Blend Shapes that is published in SIGGRAPH 2021.

Code for Learning Manifold Patch-Based Representations of Man-Made Shapes, in ICLR 2021.
Code for Learning Manifold Patch-Based Representations of Man-Made Shapes, in ICLR 2021.

LearningPatches | Webpage | Paper | Video Learning Manifold Patch-Based Representations of Man-Made Shapes Dmitriy Smirnov, Mikhail Bessmeltsev, Justi

Behind the Curtain: Learning Occluded Shapes for 3D Object Detection

Behind the Curtain: Learning Occluded Shapes for 3D Object Detection Acknowledgement We implement our model, BtcDet, based on [OpenPcdet 0.3.0]. Insta

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

CCAFNet: Crossflow and Cross-scale Adaptive Fusion Network for Detecting Salient Objects in RGB-D Images
CCAFNet: Crossflow and Cross-scale Adaptive Fusion Network for Detecting Salient Objects in RGB-D Images

Code and result about CCAFNet(IEEE TMM) 'CCAFNet: Crossflow and Cross-scale Adaptive Fusion Network for Detecting Salient Objects in RGB-D Images' IEE

Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX

ONNX msg_chn_wacv20 depth completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20 model in

Comments
  • Missing ScanNet holdout_scene images

    Missing ScanNet holdout_scene images

    There is no image data in symmetrynet/data/scannet/rgbd/holdout_scene/<scene_folder>.

    FileNotFoundError: [Errno 2] No such file or directory: '/home/zz/dataset/symmetrynet/data/scannet/rgbd/holdout_scene/scene0011_00/000000-color.jpg'
    
    opened by zzilch 3
  • What algorithm/project did you use to generate the symmetry GT data

    What algorithm/project did you use to generate the symmetry GT data

    I read your paper with great interest. It is very detailed, visual, and mathematically thorough. But one point you breeze past is the method how you generate the ground truth symmetry data. Could you explain which algorithm or project you used, so this work can be applied to other datasets like linemod.

    opened by kochsebastian 1
  • About the data set

    About the data set

    I am very interested in your work. Recently, I have tried to reproduce your work, but I do not have a complete data set. I hope I can get a complete data set.My email is [email protected] Thank you very much!

    opened by XDUWQ 5
Owner
null
(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

Bae, Gwangbin 138 Dec 28, 2022
OcclusionFusion: realtime dynamic 3D reconstruction based on single-view RGB-D

OcclusionFusion (CVPR'2022) Project Page | Paper | Video Overview This repository contains the code for the CVPR 2022 paper OcclusionFusion, where we

Wenbin Lin 193 Dec 15, 2022
Making Structure-from-Motion (COLMAP) more robust to symmetries and duplicated structures

SfM disambiguation with COLMAP About Structure-from-Motion generally fails when the scene exhibits symmetries and duplicated structures. In this repos

Computer Vision and Geometry Lab 193 Dec 26, 2022
An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

SVM Données Une base d’images contient 490 images pour l’apprentissage (400 voitures et 90 bateaux), et encore 21 images pour fait des tests. Prétrait

Achraf Rahouti 3 Nov 30, 2021
3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans.

3DMV 3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans. This work is based on our ECCV'18 p

Владислав Молодцов 0 Feb 6, 2022
DSAC* for Visual Camera Re-Localization (RGB or RGB-D)

DSAC* for Visual Camera Re-Localization (RGB or RGB-D) Introduction Installation Data Structure Supported Datasets 7Scenes 12Scenes Cambridge Landmark

Visual Learning Lab 143 Dec 22, 2022
Blender add-on: Add to Cameras menu: View → Camera, View → Add Camera, Camera → View, Previous Camera, Next Camera

Blender add-on: Camera additions In 3D view, it adds these actions to the View|Cameras menu: View → Camera : set the current camera to the 3D view Vie

German Bauer 11 Feb 8, 2022
Toward Realistic Single-View 3D Object Reconstruction with Unsupervised Learning from Multiple Images (ICCV 2021)

Table of Content Introduction Getting Started Datasets Installation Experiments Training & Testing Pretrained models Texture fine-tuning Demo Toward R

VinAI Research 42 Dec 5, 2022
PanopticBEV - Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images This r

null 63 Dec 16, 2022
Learning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image

NonCuboidRoom Paper Learning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image Cheng Yang*, Jia Zheng*, Xili Dai, Rui Tang, Yi Ma, Xiao

null 67 Dec 15, 2022