PyTorch implementations for our SIGGRAPH 2021 paper: Editable Free-viewpoint Video Using a Layered Neural Representation.

Related tags

Deep Learning st-nerf
Overview

st-nerf

We provide PyTorch implementations for our paper: Editable Free-viewpoint Video Using a Layered Neural Representation

SIGGRAPH 2021

Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu and Jingyi Yu

st-nerf: Project | Paper

Getting Started

Installation

  • Clone this repo:
git clone https://github.com/DarlingHang/st-nerf
cd st-nerf
  • Install PyTorch and other dependencies using:
conda create -n st-nerf python=3.8.5
conda activate st-nerf    
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch
conda install imageio matplotlib
pip install yacs kornia robpy

Datasets

The walking and taekwondo datasets can be downloaded from here.

Apply a pre-trained model to render demo videos

  • We provide our pretrained models which can be found under the outputs folder.
  • We provide some example scripts under the demo folder.
  • To run our demo scripts, you need to first downloaded the corresponding dataset, and put them under the folder specified by DATASETS -> TRAIN in configs/config_taekwondo.yml and configs/config_walking.yml
  • For the walking sequence, you can render videos where some performers are hided by typing the command:
python demo/walking_demo.py -c configs/config_taekwondo.yml
  • For the taekwondo sequence, you can render videos where performers are translated and scaled by typing the command:
python demo/taekwondo_demo.py -c configs/config_walking.yml
  • The rendered images and videos will be under outputs/taekwondo/rendered and outputs/walking/rendered

Acknowlegements

We borrowed some codes from Multi-view Neural Human Rendering (NHR).

Citation

If you use this code for your research, please cite our papers.

 @inproceedings{zhang2021stnerf,
                title={Editable Free-Viewpoint Video using a Layered Neural Representation},
                author={Jiakai, Zhang
                        and Xinhang, Liu
                        and Xinyi, Ye
                        and Fuqiang, Zhao
                        and Yanshun, Zhang
                        and Minye, Wu
                        and Yingliang, Zhang
                        and Lan, Xu
                        and Jingyi, Yu
                        },
                year={2021},
                booktitle={ACM SIGGRAPH},
               }
Comments
  • Evaluation results are different from the paper

    Evaluation results are different from the paper

    Hello authors, thanks for your pretrained model. I render the images of walking and ｔａｅｋｗｏｎｄｏ sequences and calculate the PSNR and SSIM metrics.

    ||PSNR|SSIM| |----|----|----| |walking | 22.8 | 0.91 | |taekwondo | 29.3 | 0.95 |

    They are slightly different from your paper. Is that normal?

    opened by chingswy 13
  • GPU RAM exhaustion!!

    GPU RAM exhaustion!!

    @DarlingHang what is minimum RAM of GPU required for the project as the process is being killed in my system due to RAM exhaustion.?? Please Guide me..

    opened by vaibhawkhemka 6
  • test error

    test error

    python demo/taekwondo_demo.py -c configs/config_taekwondo.yml Traceback (most recent call last): File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\platform\egl.py", line 67, in EGL return ctypesloader.loadLibrary( File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\platform\ctypesloader.py", line 45, in loadLibrary return dllType( name, mode ) File "D:\D\anaconda\envs\st-nerf\lib\ctypes_init_.py", line 373, in init self._handle = _dlopen(self._name, mode) FileNotFoundError: ("Could not find module 'EGL' (or one of its dependencies). Try using the full path with constructor syntax.", 'EGL', None)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "demo/taekwondo_demo.py", line 23, in from render import LayeredNeuralRenderer File ".\render_init_.py", line 4, in from .neural_renderer import NeuralRenderer File ".\render\neural_renderer.py", line 9, in from .bkgd_renderer import PrRender File ".\render\bkgd_renderer.py", line 7, in import pyrender as pr File "D:\D\anaconda\envs\st-nerf\lib\site-packages\pyrender_init_.py", line 3, in from .light import Light, PointLight, DirectionalLight, SpotLight File "D:\D\anaconda\envs\st-nerf\lib\site-packages\pyrender\light.py", line 10, in from OpenGL.GL import * File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\GL_init_.py", line 3, in from OpenGL import error as _error File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\error.py", line 12, in from OpenGL import platform, configflags File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\platform_init.py", line 35, in load() File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\platform_init.py", line 32, in _load plugin.install(globals()) File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\platform\baseplatform.py", line 92, in install namespace[ name ] = getattr(self,name,None) File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\platform\baseplatform.py", line 14, in get value = self.fget( obj ) File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\platform\egl.py", line 93, in GetCurrentContext return self.EGL.eglGetCurrentContext File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\platform\baseplatform.py", line 14, in get value = self.fget( obj ) File "D:\D\anaconda\envs\st-nerf\lib\site-packages\OpenGL\platform\egl.py", line 73, in EGL raise ImportError("Unable to load EGL library", *err.args) ImportError: ('Unable to load EGL library', "Could not find module 'EGL' (or one of its dependencies). Try using the full path with constructor syntax.", 'EGL', None)

    platform: win10

    opened by ChaoyiXie 3
  • can't download the demo dataset

    can't download the demo dataset

    The link https://drive.google.com/drive/folders/13YHw_YSGewvcgYdwqbelM9L2JiPNWLi7 seems empty, and I can't download the dataset. Could you please update the link and share the dataset? Thanks very much!

    opened by sycamore27 1
  • Problem with torchsearchsorted

    Problem with torchsearchsorted

    Hello,

    I cannot install the torchsearchsorted package, the following error is returned :

    Failed to build torchsearchsorted Installing collected packages: torchsearchsorted Running setup.py install for torchsearchsorted ... error

    Do you know what the error might be?

    opened by FaizSofiane 1
  • test error

    test error

    RuntimeError:Tried to access nonexistent attribute or method 'call.' of type "'.torch.-.konia.geometry.boxes.Boxes3D'. Did you forget to initialize an attribute in init()?: File "E:\Anaconda3\envs\st-nerf\lib\site-packages\kornia\geometry\boxes.py"，line 718# Due to some torch.jit.script bug (at least <= 1.9), you need to pass all arguments to .-init__ when# constructing the class from inside of a method. return cls(hexahedrons, raise if_not floating point=False mode=mode)222 <--- HERE

    Hello, the following errors occur when I run the test code. How can I solve them

    opened by lzsupupup 0
  • Rotation or Duplication doesn't work or exist.

    Rotation or Duplication doesn't work or exist.

    I wanted to know whether you didn't implement the rotation and the duplication of the datasets, because I didn't get it to work. I tried to set the Rotation in the init-method but it didn't change anything.

    opened by xherdon 0
  • Questions about the synthetic dataset.

    Questions about the synthetic dataset.

    I am confused by your synthetic dataset:

    1. How do you generated the ground-truth frames for the 36 virtual cameras?
    2. Why does the number of virtual cameras changes from 36 to 17 (Tab. 3)?
    opened by FreemanG 0
Owner
Diplodocus
Diplodocus
GitHub
Layered Neural Atlases for Consistent Video Editing

Layered Neural Atlases for Consistent Video Editing Project Page | Paper This repository contains an implementation for the SIGGRAPH Asia 2021 paper L

Yoni Kasten 353 Dec 27, 2022
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

null 42 Nov 24, 2022
A self-supervised 3D representation learning framework named viewpoint bottleneck.

Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck Paper Created by Liyi Luo, Beiwen Tian, Hao Zhao and Guyue Zhou from Institute for AI In

null 63 Aug 11, 2022
A self-supervised 3D representation learning framework named viewpoint bottleneck.

Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck Paper Created by Liyi Luo, Beiwen Tian, Hao Zhao and Guyue Zhou from Institute for AI In

null 42 Sep 24, 2021
A supplementary code for Editable Neural Networks, an ICLR 2020 submission.

Editable neural networks A supplementary code for Editable Neural Networks, an ICLR 2020 submission by Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Py

Anton Sinitsin 32 Nov 29, 2022
ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021

ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021 Dataset Code Demos Authors: He Zhang, Yuting Ye, Tak

HE ZHANG 194 Dec 6, 2022
Eff video representation - Efficient video representation through neural fields

Neural Residual Flow Fields for Efficient Video Representations 1. Download MPI

null 41 Jan 6, 2023
This is the open-source reference implementation of the SIGGRAPH 2021 paper Intersection-free Rigid Body Dynamics.

Robust, intersection-free, simulations of rigid bodies.

null 93 Dec 23, 2022
Code reproduce for paper "Vehicle Re-identification with Viewpoint-aware Metric Learning"

VANET Code reproduce for paper "Vehicle Re-identification with Viewpoint-aware Metric Learning" Introduction This is the implementation of article VAN

EMDATA-AILAB 23 Dec 26, 2022
Code for "Unsupervised Layered Image Decomposition into Object Prototypes" paper

DTI-Sprites Pytorch implementation of "Unsupervised Layered Image Decomposition into Object Prototypes" paper Check out our paper and webpage for deta

null 40 Dec 22, 2022
Code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in Video".

Consistent Depth of Moving Objects in Video This repository contains training code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in

Google 203 Jan 5, 2023
This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

The Neural Process Family This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CN

DeepMind 892 Dec 28, 2022
Efficient electromagnetic solver based on rigorous coupled-wave analysis for 3D and 2D multi-layered structures with in-plane periodicity

Efficient electromagnetic solver based on rigorous coupled-wave analysis for 3D and 2D multi-layered structures with in-plane periodicity, such as gratings, photonic-crystal slabs, metasurfaces, surface-emitting lasers, nano-antennas, and more.

Alex Song 17 Dec 19, 2022
Read and write layered TIFF ImageSourceData and ImageResources tags

Read and write layered TIFF ImageSourceData and ImageResources tags Psdtags is a Python library to read and write the Adobe Photoshop(r) specific Imag

Christoph Gohlke 4 Feb 5, 2022
[CVPR2021] The source code for our paper 《Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning》.

TBE The source code for our paper "Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Le

Jinpeng Wang 150 Dec 28, 2022
[AAAI2021] The source code for our paper 《Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion》.

DSM The source code for paper Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion Project Website; Datasets li

Jinpeng Wang 114 Oct 16, 2022
PyTorch implementations of the paper: "DR.VIC: Decomposition and Reasoning for Video Individual Counting, CVPR, 2022"

DRNet for Video Indvidual Counting (CVPR 2022) Introduction This is the official PyTorch implementation of paper: DR.VIC: Decomposition and Reasoning

tao han 35 Nov 22, 2022
Unofficial pytorch implementation of paper "One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing"

One-Shot Free-View Neural Talking Head Synthesis Unofficial pytorch implementation of paper "One-Shot Free-View Neural Talking-Head Synthesis for Vide

ZLH 406 Dec 23, 2022
Supplementary code for SIGGRAPH 2021 paper: Discovering Diverse Athletic Jumping Strategies

SIGGRAPH 2021: Discovering Diverse Athletic Jumping Strategies project page paper demo video Prerequisites Important Notes We suspect there are bugs i

null 54 Dec 6, 2022