Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian (CVPR 2022)

Overview

Pop-Out Motion

Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian (CVPR 2022)

Jihyun Lee*, Minhyuk Sung*, Hyunjin Kim, Tae-Kyun (T-K) Kim (*: equal contributions)

[Project Page] [Paper] [Video]

animated

We present a framework that can deform an object in a 2D image as it exists in 3D space. While our method leverages 2D-to-3D reconstruction, we argue that reconstruction is not sufficient for realistic deformations due to the vulnerability to topological errors. Thus, we propose to take a supervised learning-based approach to predict the shape Laplacian of the underlying volume of a 3D reconstruction represented as a point cloud. Given the deformation energy calculated using the predicted shape Laplacian and user-defined deformation handles (e.g., keypoints), we obtain bounded biharmonic weights to model plausible handle-based image deformation.

 

Environment Setup

Clone this repository and install the dependencies specified in requirements.txt.

 git clone https://github.com/jyunlee/Pop-Out-Motion.git
 mv Pop-Out-Motion
 pip install -r requirements.txt 

 

Data Pre-Processing

Training Data

  1. Build executables from the c++ files in data_preprocessing directory. After running the commands below, you should have normalize_bin and calc_l_minv_bin executables.
 cd data_preprocessing
 mkdir build
 cd build
 cmake ..
 make
 cd ..
  1. Clone and build Manifold repository to obtain manifold executable.

  2. Clone and build fTetWild repository to obtain FloatTetwild_bin executable.

  3. Run preprocess_train_data.py to prepare your training data. This should perform (1) shape normalization into a unit bounding sphere, (2) volume mesh conversion, and (3) cotangent Laplacian and inverse mass calculation.

 python preprocess_train_data.py 

Test Data

  1. Build executables from the c++ files in data_preprocessing directory. After running the commands below, you should have normalize_bin executable.
 cd data_preprocessing
 mkdir build
 cd build
 cmake ..
 make
 cd ..
  1. Run preprocess_test_data.py to prepare your test data. This should perform (1) shape normalization into a unit bounding sphere and (2) pre-computation of KNN-Based Point Pair Sampling (KPS).
 python preprocess_test_data.py 

 

Network Training

Run network/train.py to train your own Laplacian Learning Network.

 cd network
 python train.py 

The pre-trained model on DFAUST dataset is also available here.

 

Network Inference

Deformation Energy Inference

  1. Given an input image, generate its 3D reconstruction via running PIFu. It is also possible to directly use point cloud data obtained from other sources.

  2. Pre-process the data obtained from Step 1 -- please refer to this section.

  3. Run network/a_inference.py to predict the deformation energy matrix.

 cd network
 python a_inference.py 

Handle-Based Deformation Weight Calculation

  1. Build an executable from the c++ file in bbw_calculation directory. After running the commands below, you should have calc_bbw_bin executable.
 cd bbw_calculation
 mkdir build
 cd build
 cmake ..
 make
 cd ..
  1. (Optional) Run sample_pt_handles.py to obtain deformation control handles sampled by farthest point sampling.

  2. Run calc_bbw_bin to calculate handle-based deformation weights using the predicted deformation energy.

./build/calc_bbw_bin <shape_path> <handle_path> <deformation_energy_path> <output_weight_path>

 

Citation

If you find this work useful, please consider citing our paper.

@InProceedings{lee2022popoutmotion,
    author = {Lee, Jihyun and Sung, Minhyuk and Kim, Hyunjin and Kim, Tae-Kyun},
    title = {Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2022}
}

 

Acknowledgements

You might also like...
Motion and Shape Capture from Sparse Markers

MoSh++ This repository contains the official chumpy implementation of mocap body solver used for AMASS: AMASS: Archive of Motion Capture as Surface Sh

Code for 'Single Image 3D Shape Retrieval via Cross-Modal Instance and Category Contrastive Learning', ICCV 2021
Code for 'Single Image 3D Shape Retrieval via Cross-Modal Instance and Category Contrastive Learning', ICCV 2021

CMIC-Retrieval Code for Single Image 3D Shape Retrieval via Cross-Modal Instance and Category Contrastive Learning. ICCV 2021. Introduction In this wo

The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.
The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.

NTIRE 2022 - Image Inpainting Challenge Important dates 2022.02.01: Release of train data (input and output images) and validation data (only input) 2

Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images

SASSnet Code for paper: Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images(MICCAI 2020) Our code is origin from UA-MT You can fin

PyTorch implementation of ShapeConv: Shape-aware Convolutional Layer for RGB-D Indoor Semantic Segmentation.
PyTorch implementation of ShapeConv: Shape-aware Convolutional Layer for RGB-D Indoor Semantic Segmentation.

Shape-aware Convolutional Layer (ShapeConv) PyTorch implementation of ShapeConv: Shape-aware Convolutional Layer for RGB-D Indoor Semantic Segmentatio

ICCV2021 Paper: AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection
ICCV2021 Paper: AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection

ICCV2021 Paper: AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection

PyTorch implementation of
PyTorch implementation of "Representing Shape Collections with Alignment-Aware Linear Models" paper.

deep-linear-shapes PyTorch implementation of "Representing Shape Collections with Alignment-Aware Linear Models" paper. If you find this code useful i

Code for
Code for "Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks", CVPR 2021

Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks This repository contains the code that accompanies our CVPR 20

Code for
Code for "LASR: Learning Articulated Shape Reconstruction from a Monocular Video". CVPR 2021.

LASR Installation Build with conda conda env create -f lasr.yml conda activate lasr # install softras cd third_party/softras; python setup.py install;

Comments
  • Could you share the PIFu model for the character images?

    Could you share the PIFu model for the character images?

    Thank you for this great work! I am wondering whether it is possible to release the trained PIFu model for the character images. Your help is greatly appreciated!

    opened by jind11 1
  • Fail to build data_preprocessing

    Fail to build data_preprocessing

    Thanks for your great work! I encounter some problems when building executables from the c++ files in data_preprocessing.

    CMake Error at Pop-Out-Motion/libigl/cmake/libigl.cmake:5 (message): You included libigl.cmake directly from your own project. This behavior is not supported anymore. Please add libigl to your project via add_subdirectory(<path_to_libigl>). See the libigl example project for more information: https://github.com/libigl/libigl-example-project/ Call Stack (most recent call first): cmake/FindLIBIGL.cmake:37 (include) CMakeLists.txt:10 (find_package)

    However, I build the libigl-example-project successfully. Could you provide some solutions?

    opened by zyhbili 1
Owner
Jihyun Lee
Jihyun Lee
The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Neural Deformation Graphs Project Page | Paper | Video Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction Aljaž Božič, Pablo P

Aljaz Bozic 130 Sep 7, 2022
Joint Learning of 3D Shape Retrieval and Deformation, CVPR 2021

Joint Learning of 3D Shape Retrieval and Deformation Joint Learning of 3D Shape Retrieval and Deformation Mikaela Angelina Uy, Vladimir G. Kim, Minhyu

Mikaela Uy 34 Aug 25, 2022
A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022)

A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022) https://arxiv.org/abs/2203.09388 Jianqi Ma, Zheto

MA Jianqi, shiki 89 Sep 24, 2022
DeepMetaHandles: Learning Deformation Meta-Handles of 3D Meshes with Biharmonic Coordinates

DeepMetaHandles (CVPR2021 Oral) [paper] [animations] DeepMetaHandles is a shape deformation technique. It learns a set of meta-handles for each given

Liu Minghua 71 Sep 29, 2022
Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals

LapDepth-release This repository is a Pytorch implementation of the paper "Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals" M

Minsoo Song 181 Sep 21, 2022
[CVPR 2022] Back To Reality: Weak-supervised 3D Object Detection with Shape-guided Label Enhancement

Back To Reality: Weak-supervised 3D Object Detection with Shape-guided Label Enhancement Announcement ?? We have not tested the code yet. We will fini

Xiuwei Xu 6 Mar 30, 2022
This repository contains the code for the paper "Hierarchical Motion Understanding via Motion Programs"

Hierarchical Motion Understanding via Motion Programs (CVPR 2021) This repository contains the official implementation of: Hierarchical Motion Underst

Sumith Kulal 38 Aug 16, 2022
Exploring Versatile Prior for Human Motion via Motion Frequency Guidance (3DV2021)

Exploring Versatile Prior for Human Motion via Motion Frequency Guidance This is the codebase for video-based human motion reconstruction in human-mot

Jiachen Xu 5 Jul 14, 2022
[CVPR 2022] CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation

CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation Prerequisite Please create and activate the following conda envrionment. To r

Qin Wang 73 Sep 22, 2022
Official pytorch implementation for Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion (CVPR 2022)

Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion This repository contains a pytorch implementation of "Learning to Listen: Modeling

null 41 Aug 14, 2022