SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes
Paper | Supp | Video | Project Page | Blog (AIT，AVG)
Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes. We propose a novel forward skinning module to animate neural implicit shapes with good generalization to unseen poses.
If you find our code or paper useful, please cite as
@inproceedings{chen2021snarf,
title={SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes},
author={Chen, Xu and Zheng, Yufeng and Black, Michael J and Hilliges, Otmar and Geiger, Andreas},
booktitle={International Conference on Computer Vision (ICCV)},
year={2021}
}
Quick Start
Clone this repo:
git clone https://github.com/xuchen-ethz/snarf.git
cd snarf
Install environment:
conda env create -f environment.yml
conda activate snarf
python setup.py install
Download SMPL models (1.0.0 for Python 2.7 (10 shape PCs)) and move them to the corresponding places:
mkdir lib/smpl/smpl_model/
mv /path/to/smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl lib/smpl/smpl_model/SMPL_FEMALE.pkl
mv /path/to/smpl/models/basicmodel_m_lbs_10_207_0_v1.0.0.pkl lib/smpl/smpl_model/SMPL_MALE.pkl
Download our pretrained models and test motion sequences:
sh ./download_data.sh
Run a quick demo for clothed human:
python demo.py expname=cape subject=3375 demo.motion_path=data/aist_demo/seqs +experiments=cape
You can the find the video in outputs/cape/3375/demo.mp4 and images in outputs/cape/3375/images/. To save the meshes, add demo.save_mesh=true to the command.
You can also try other subjects (see outputs/data/cape for available options) by setting subject=xx, and other motion sequences from AMASS by setting demo.motion_path=/path/to/amass_modetion.npz.
Some motion sequences have high fps and one might want to skip some frames. To do this, add demo.every_n_frames=x to consider every x frame in the motion sequence. (e.g. demo.every_n_frames=10 for PosePrior sequences)
By default, we use demo.fast_mode=true for fast mesh extraction. In this mode, we first extract mesh in canonical space, and then forward skin the mesh to posed space. This bypasses the root finding during inference, thus is faster. However, it's not really deforming a continuous field. To first deform the continuous field and then extract mesh in deformed space, use demo.fast_mode=false instead.
Training and Evaluation
Install Additional Dependencies
Install kaolin for fast occupancy query from meshes.
git clone https://github.com/NVIDIAGameWorks/kaolin
cd kaolin
git checkout v0.9.0
python setup.py develop
Minimally Clothed Human
Prepare Datasets
Download the AMASS dataset. We use ''DFaust Snythetic'' and ''PosePrior'' subsets and SMPL-H format. Unzip the dataset into data folder.
tar -xf DFaust67.tar.bz2 -C data
tar -xf MPILimits.tar.bz2 -C data
Preprocess dataset:
python preprocess/sample_points.py --output_folder data/DFaust_processed
python preprocess/sample_points.py --output_folder data/MPI_processed --skip 10 --poseprior
Training
Run the following command to train for a specified subject:
python train.py subject=50002
Training logs are available on wandb (registration needed, free of charge). It should take ~12h on a single 2080Ti.
Evaluation
Run the following command to evaluate the method for a specified subject on within distribution data (DFaust test split):
python test.py subject=50002
and outside destribution (PosePrior):
python test.py subject=50002 datamodule=jointlim
Generate Animation
You can use the trained model to generate animation (same as in Quick Start):
python demo.py expname='dfaust' subject=50002 demo.motion_path='data/aist_demo/seqs'
Clothed Human
Training
Download the CAPE dataset and unzip into data folder.
Run the following command to train for a specified subject and clothing type:
python train.py datamodule=cape subject=3375 datamodule.clothing='blazerlong' +experiments=cape
Training logs are available on wandb. It should take ~24h on a single 2080Ti.
Generate Animation
You can use the trained model to generate animation (same as in Quick Start):
python demo.py expname=cape subject=3375 demo.motion_path=data/aist_demo/seqs +experiments=cape
Acknowledgement
We use the pre-processing code in PTF and LEAP with some adaptions (./preprocess). The network and sampling part of the code (lib/model/network.py and lib/model/sample.py) is implemented based on IGR and IDR. The code for extracting mesh (lib/utils/meshing.py) is adapted from NASA. Our implementation of Broyden's method (lib/model/broyden.py) is based on DEQ. We sincerely thank these authors for their awesome work.
109 Nov 22, 2022
296 Dec 29, 2022
132 Nov 28, 2022
442 Dec 16, 2022
302 Dec 14, 2022
66 Dec 26, 2022
1.5k Jan 2, 2023
684 Dec 26, 2022
45 Oct 7, 2022
697 Jan 6, 2023
144 Jan 4, 2023
14 Nov 21, 2022
16 Oct 5, 2022
338 Dec 28, 2022
80 Dec 8, 2022
157 Dec 26, 2022
170 Dec 27, 2022
93 Dec 23, 2022