[3DV 2020] PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction

Overview

PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction

International Conference on 3D Vision, 2020

1Center for Visual Information Technology, IIIT Hyderabad



Abstract
We introduce PeeledHuman - a novel shape representation of the human body that is robust to self-occlusions. PeeledHuman encodes the human body as a set of Peeled Depth and RGB maps in 2D, obtained by performing raytracing on the 3D body model and extending each ray beyond its first intersection. This formulation allows us to handle self-occlusions efficiently compared to other representations. Given a monocular RGB image, we learn these Peeled maps in an end-to-end generative adversarial fashion using our novel framework - PeelGAN. We train PeelGAN using a 3D Chamfer loss and other 2D losses to generate multiple depth values per-pixel and a corresponding RGB field per-vertex in a dual-branch setup. In our simple non-parametric solution, the generated Peeled Depth maps are back-projected to 3D space to obtain a complete textured 3D shape. The corresponding RGB maps provide vertex-level texture details. We compare our method with current parametric and non-parametric methods in 3D reconstruction and find that we achieve state-of-theart-results. We demonstrate the effectiveness of our representation on publicly available BUFF and MonoPerfCap datasets as well as loose clothing data collected by our calibrated multi-Kinect setup.

Testing

Install environment

$ conda env create -f environment.yml

Download the checkpoint from here and store it in ./checkpoints/test/. The provided checkpoint was trained on the MonoPerfCap dataset.

Run the inference script

python test.py                            \
  --test_folder_path <path/to/images/dir> \
  --results_dir <path/to/results/dir>     \
  --name test                             \
  --direction AtoB                        \
  --model pix2pix                         \
  --netG resnet_18blocks                  \
  --output_nc 4                           \
  --load_size 512                         \
  --eval

The script looks for the checkpoint file in checkpoints/<checkpoint/name>

Citation

@inproceedings {jinka2020peeledhuman,
  author = {S. Jinka and R. Chacko and A. Sharma and P. Narayanan},
  booktitle = {2020 International Conference on 3D Vision (3DV)},
  title = {PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction},
  year = {2020},
  pages = {879-888},
  doi = {10.1109/3DV50981.2020.00098},
  publisher = {IEEE Computer Society},
  }
  

Acknowledgements

Our network derives from the pix2pix work and hence builds on the official PyTorch implementation of pix2pix. This README template was borrowed from Aakash Kt. Please open an issue in case of any bugs/queries.

You might also like...
HPRNet: Hierarchical Point Regression for Whole-Body Human Pose Estimation

HPRNet: Hierarchical Point Regression for Whole-Body Human Pose Estimation Official PyTroch implementation of HPRNet. HPRNet: Hierarchical Point Regre

Ultra-lightweight human body posture key point CNN model. ModelSize:2.3MB  HUAWEI P40 NCNN benchmark: 6ms/img,
Ultra-lightweight human body posture key point CNN model. ModelSize:2.3MB HUAWEI P40 NCNN benchmark: 6ms/img,

Ultralight-SimplePose Support NCNN mobile terminal deployment Based on MXNET(=1.5.1) GLUON(=0.7.0) framework Top-down strategy: The input image is t

 SMPL-X: A new joint 3D model of the human body, face and hands together
SMPL-X: A new joint 3D model of the human body, face and hands together

SMPL-X: A new joint 3D model of the human body, face and hands together [Paper Page] [Paper] [Supp. Mat.] Table of Contents License Description News I

Face and Pose detector that emits MQTT events when a face or human body is detected and not detected.
Face and Pose detector that emits MQTT events when a face or human body is detected and not detected.

Face Detect MQTT Face or Pose detector that emits MQTT events when a face or human body is detected and not detected. I built this as an alternative t

A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.
A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.

ManhattanSLAM Authors: Raza Yunus, Yanyan Li and Federico Tombari ManhattanSLAM is a real-time SLAM library for RGB-D cameras that computes the camera

"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022) Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Z

Public repository of the 3DV 2021 paper "Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds"

Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds Björn Michele1), Alexandre Boulch1), Gilles Puy1), Maxime Bucher1) and Rena

RGBD-Net - This repository contains a pytorch lightning implementation for the 3DV 2021 RGBD-Net paper.
RGBD-Net - This repository contains a pytorch lightning implementation for the 3DV 2021 RGBD-Net paper.

[3DV 2021] We propose a new cascaded architecture for novel view synthesis, called RGBD-Net, which consists of two core components: a hierarchical depth regression network and a depth-aware generator network.

Code for
Code for "3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop"

PyMAF This repository contains the code for the following paper: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop Hongwe

Comments
  • NameError: name 'convert_model' is not defined

    NameError: name 'convert_model' is not defined

    Traceback (most recent call last):
      File "test.py", line 119, in <module>
        model = create_model(opt)   # create a model given opt.model and other options
      File "/media/fsa/DATA/Work/GIST/PeeledHuman/models/__init__.py", line 68, in create_model
        instance = model(opt)
      File "/media/fsa/DATA/Work/GIST/PeeledHuman/models/pix2pix_model.py", line 50, in __init__
        not opt.no_dropout, opt.init_type, opt.init_gain, self.gpu_ids)
      File "/media/fsa/DATA/Work/GIST/PeeledHuman/models/networks.py", line 149, in define_G
        return init_net(net, init_type, init_gain, gpu_ids)
      File "/media/fsa/DATA/Work/GIST/PeeledHuman/models/networks.py", line 115, in init_net
        net = convert_model(net)
    NameError: name 'convert_model' is not defined
    
    
    opened by jaymefosa 1
  • Code not working

    Code not working

    Code references a custom_dataset module that doesn't exist. Removed it. Then I get this ghost parameter

    test.py: error: the following arguments are required: --dataroot

    There seems to be no description to what this parameters is for. Passing it in blank we get

    TypeError: Can't instantiate abstract class Pix2PixModel with abstract methods optimize_parameters

    Did anyone work through all of this to get the base pretrained model working?

    opened by chrisdottel 1
  • video Inference?

    video Inference?

    Hello, thank you for your great work How to run the inference on a Video? Can you provide a colab demo? It will help everyone and will be much appreciated thank you

    opened by Adeel-Intizar 0
Owner
Rohan Chacko
CS Senior | Working on 3D Human reconstruction
Rohan Chacko
Learning Generative Models of Textured 3D Meshes from Real-World Images, ICCV 2021

Learning Generative Models of Textured 3D Meshes from Real-World Images This is the reference implementation of "Learning Generative Models of Texture

Dario Pavllo 115 Jan 7, 2023
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors, CVPR 2021

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors Human POSEitioning System (H

Aymen Mir 66 Dec 21, 2022
Expressive Body Capture: 3D Hands, Face, and Body from a Single Image

Expressive Body Capture: 3D Hands, Face, and Body from a Single Image [Project Page] [Paper] [Supp. Mat.] Table of Contents License Description Fittin

Vassilis Choutas 1.3k Jan 7, 2023
Full body anonymization - Realistic Full-Body Anonymization with Surface-Guided GANs

Realistic Full-Body Anonymization with Surface-Guided GANs This is the official

Håkon Hukkelås 30 Nov 18, 2022
WormMovementSimulation - 3D Simulation of Worm Body Movement with Neurons attached to its body

Generate 3D Locomotion Data This module is intended to create 2D video trajector

null 1 Aug 9, 2022
From a body shape, infer the anatomic skeleton.

OSSO: Obtaining Skeletal Shape from Outside (CVPR 2022) This repository contains the official implementation of the skeleton inference from: OSSO: Obt

Marilyn Keller 166 Dec 28, 2022
[WACV 2020] Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints

Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints Official implementation for Reducing Footskate in Human Motion Recon

Virginia Tech Vision and Learning Lab 38 Nov 1, 2022
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

null 42 Nov 24, 2022
Code for "LASR: Learning Articulated Shape Reconstruction from a Monocular Video". CVPR 2021.

LASR Installation Build with conda conda env create -f lasr.yml conda activate lasr # install softras cd third_party/softras; python setup.py install;

Google 157 Dec 26, 2022
ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction

ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction. NeurIPS 2021.

Gengshan Yang 59 Nov 25, 2022