Editing a Conditional Radiance Field

Overview

Editing Conditional Radiance Fields

Project | Paper | Video | Demo

Editing Conditional Radiance Fields
Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, Bryan Russell
MIT, Adobe Research, CMU
in arXiv:2105.06466, 2021.

Editing Results

Color Editing


Our method propagates sparse 2D user scribbles to fill an object region, rendering the edit consistently across views. The user provides a color, a foreground scribble for the region to change, and a background scribble for regions to keep unchanged. To conduct the edit, we optimize a reconstruction-based loss to encourage the model to change the color at the foreground scribble, but maintain the color on the background scribbles.

Shape Editing


Our method propagates 2D user edits to remove or add an object part, propagating the 2D edit consistently across views. For shape removal, the user scribbles over a region of the object to remove. To conduct the removal, we optimize both a reconstruction loss and a density-based loss, encouraging the model to remove density at the scribbled regions. For shape addition, the user selects an object part to paste into the instance. To conduct the addition, we optimize a reconstruction loss similar to the one used for color editing.

Color and Shape Transfer


Our method can transfer shape and color between object instances simply by swapping the color and shape codes between instances.

Editing a Real Image


Our method is able to render novel views of the real object instance and conduct color and shape editing on the instance.

Method


To propagate sparse 2D user scribbles to novel views, we learn a rich prior of plausible-looking objects by training a single radiance field over several object instances. Our architecture builds on NeRF in two ways. First, we introduce shape and color codes for each instance, allowing a single radiance field to represent multiple object instances. Second, we introduce an instance independent shape branch, which learns a generic representation of the object category. Due to our modular architecture design, only a few components of our network need to be modified during editing to effectively execute the user edit.

Getting Started

Installation

  • Clone this repo:
git clone https://github.com/stevliu/editnerf.git
cd editnerf
  • Install the dependencies
bash scripts/setup_env.sh
  • Obtain pre-trained models and editing examples:
bash scripts/setup_models.sh
  • Optionally, download the relevant datasets. This step is required to evaluate edits and for training/testing a conditional radiance field:
bash scripts/setup_data.sh

Our code is tested on using Python 3.6, PyTorch 1.3.1, and CUDA 10.1.

Editing a Conditional Radiance Field

To conduct your own edits, please check out our demo. Alternatively, you can run the demo locally using jupyter notebook and using the notebook ui/editing.ipynb.

To execute the edits used in our paper, please run:

bash scripts/editing_experiments.sh

To evaluate the edits used in our paper, please run:

bash scripts/evaluate_edits.sh

Feel free to check out additional editing examples, which can be run via scripts/additional_edits.sh.

Learning a Conditional Radiance Field

Training

To train a conditional radiance field on the PhotoShapes dataset, please run:

python run_nerf.py --config configs/photoshapes/config.txt --skip_loading

The --skip_loading flag tells the script not to load the pretrained weights during training.

To train on other datasets, or use a different model architecture, you can replace the config file with your own. Feel free to check out example config files under configs/. For additional training options, please visit inputs.py.

Evaluation

To render train and test views from a conditional radiance field, you can run:

python test_nerf.py --config config-file --render_test --render_train

where config-file is the same config file used during training.

Then, to run evaluation metrics on the rendered samples, you can run:

python utils/evaluate_reconstruction.py --expdir path-to-log-dir

To evaluate the conditional radiance fields used in our paper, please run:

bash scripts/reconstruction_experiments.sh

Training and Editing Your Own Models

To train a model on a different dataset, first setup the directory to store the dataset. The structure should be

data/
    datasetname/
        instances.txt
        instance_name1
            images
            transforms_train.json
            transforms_val.json
            trainsforms_test.json
        instance_name2
            ...
        ...

Each instance subdirectory should contain transforms_train.json, transforms_test.json, and transforms_val.json. Each of these .json files should contain the camera focal, as well as the camera extrinsics for each image. Please refer to data/photoshapes/shape09135_rank02/transforms_train.json for an example. instances.txt should contain a list of the instance names.

Then you can run python run_nerf.py --config config-file to train a conditional radiance field, and evaluate it using the instructions in the above section.

To edit the conditional radiance field, first make a directory in ui which will contain all the relevant data for the model. Then, copy over the config file, model weights, camera intrinsics, and camera extrinsics (last three are automatically saved under logs/). The directory structure should be

ui/
    datasetname/
        model.tar
        hwfs.npy
        poses.npy
        config.txt

Please refer to ui/photoshapes for an example.

Editing a Real Image

To edit a real image, we first decide on a base model to finetune to the real image. In our experiments, we use the Dosovitskiy chairs model. Then, visually estimate the pose of the image. One way to do this is by finding the nearest neighbor pose in the training set of the base model. Then, construct the dataset folder containing the .json files mentioned in the above section.

The directory structure should be

realchairname/
    images
    transforms_train.json
    transforms_val.json
    trainsforms_test.json

As an example, please refer to data/real_chairs/shape00001_charlton.

To finetune the radiance field on this image, you can run

python run_nerf.py --config base-config --real_image_dir data-dir --savedir savedir --n_iters_code_only 1000 --style_optimizer lbfgs

where base-config is the model to fit, data_dir is the directory containing the real images, and savedir is where you want to save the results. The last two flags tell the training script to first finetune the shape and color codes using LBFGS. Please refer to scripts/reconstruction_experiments.sh for an example.

To edit this instance, copy the finetuned model weights from savedir and to a subdirectory of the base model in ui. Then, copy over the camera intrinsics and camera extrinsics (located under logs/). The directory structure should be

ui/
    basemodel/
        realchair/
            model.tar
            hwfs.npy
            poses.npy

Please refer to ui/dosovitskiy_chairs/real_chair for an example.

Acknowledgments

This codebase is heavily based on the nerf-pytorch code base, and our user interface is heavily based on the GAN rewriting interface. We also use LBFGS code from PyTorch-LBFGS and job scheduling code from the GAN seeing codebase.

We thank all authors for the wonderful code!

Citation

If you use this code for your research, please cite the following work.

@misc{liu2021editing,
      title={Editing Conditional Radiance Fields},
      author={Steven Liu and Xiuming Zhang and Zhoutong Zhang and Richard Zhang and Jun-Yan Zhu and Bryan Russell},
      year={2021},
      eprint={2105.06466},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
Comments
  • Query regarding missing file

    Query regarding missing file

    Hi, thanks for sharing this code.

    When I tried to run your jupyter file, I find some file missing like there is no upsample.py in ui_utils, and the jupyter file cannot run with the default configuration.

    Any suggestions and updates?

    opened by NIRVANALAN 2
  • Cannot download pre-trained models

    Cannot download pre-trained models

    Hi, thanks for sharing this code.

    The pre-trained models cannot be downloaded from editnerf.csail.mit.edu/weights.zip when I run setup_models.sh.

    -- http://editnerf.csail.mit.edu/weights.zip Resolving editnerf.csail.mit.edu (editnerf.csail.mit.edu)... 128.52.131.63 Connecting to editnerf.csail.mit.edu (editnerf.csail.mit.edu)|128.52.131.63|:80... failed: Connection timed out. Retrying.

    Could you check whether the URL can still be accessed? Thanks!

    opened by 1180500707 1
  • Why don't you update shape code when editing shape

    Why don't you update shape code when editing shape

    As is mentioned in your paper, you update F_rad and color code z^c when editing colors, while you only update F_fuse and F_dens when editing shape. I'm wondering why don't you also update shape code z^s when editing shape.

    opened by ghy0324 1
  • How to make transforms json file for custom data

    How to make transforms json file for custom data

    Hello, I am a college student studying artificial intelligence. First, I would like to thank you for your excellent research. I would like to try custom data for this model, but I would like to ask how I can make the contents of transforms_train.json. If you know, please reply.

    opened by datu0615 0
  • How to obtain the shape code and color code of the input

    How to obtain the shape code and color code of the input

    I am very interested in your excellent work. I want to ask how to obtain the shape code and color code of the input. In the same class, such as the same chair class, different instances have different codes. Are these two codes trained before editing? How can I use labels to supervise the training of these two codes? The network parameters are obtained through training. How to obtain the input code?

    opened by ClimberY 0
  • Could it works on face editing?

    Could it works on face editing?

    Why do you want to edit objects in the first place rather than human body or face? I have seen a lot of non-human editing, is there have a big trouble in human body or face editing?

    opened by ZziTaiLeo 0
  • train style vectors from scratch

    train style vectors from scratch

    Hello,

    Thanks for sharing your wonderful work with the community.

    I have a question regarding the style vectors. In the default training code, these vectors are loaded from a pre-trained model (checkpoint = 990000.tar). Instead, to train both the NeRF and the style model from scratch, I call the run_nerf function with "--no_reload" and "--skip_loading" arguments. However, this model does not seem to be converging as well as the expected checkpoint when loaded from the pre-trained style vectors. Could you please provide instructions on how to train the model correctly from scratch (including the style vectors) without any pre-trained model weights?

    Thanks!

    opened by ishanic 0
  • Optimizer.step fails with RuntimeError: The size of tensor a (163) must match the size of tensor b (256)

    Optimizer.step fails with RuntimeError: The size of tensor a (163) must match the size of tensor b (256)

    Hi, Thank you for the great work, it was interesting to read this.

    When I try to run this for the example real image fitting as below.

    python run_nerf.py --config configs/dosovitskiy_chairs/config.txt --real_image_dir data/real_chairs/shape00001_charlton --N_rand 512 --n_iters_real 10000 --n_iters_code_only 1000 --style_optimizer lbfgs --i_testset 1000 --i_weights 1000 --savedir real_chairs/shape00001_charlton --testskip 1

    The code works until it's optimizing the code only. But then when it switches to "jointly optimize weights with code". It fails at optimizer.step with the below error.

    Starting to jointly optimize weights with code
    Traceback (most recent call last):
      File "run_nerf.py", line 210, in <module>
        train()
      File "run_nerf.py", line 125, in train
        optimizer.step()
      File "/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py", line 88, in wrapper
        return func(*args, **kwargs)
      File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
        return func(*args, **kwargs)
      File "/usr/local/lib/python3.7/dist-packages/torch/optim/adam.py", line 118, in step
        eps=group['eps'])
      File "/usr/local/lib/python3.7/dist-packages/torch/optim/_functional.py", line 86, in adam
        exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
    RuntimeError: The size of tensor a (163) must match the size of tensor b (256) at non-singleton dimension 1
    

    Further the above command works when fine when we skip loading the pre-trained model by using this flag --skip_loading.

    Any suggestions on how to go about solving this?

    opened by vishnukool 1
Cancer metastasis detection with neural conditional random field (NCRF)

NCRF Prerequisites Data Whole slide images Annotations Patch images Model Training Testing Tissue mask Probability map Tumor localization FROC evaluat

Baidu Research 731 Jan 1, 2023
This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Deformable Neural Radiance Fields This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies. Project Page Paper Video This codebase conta

Google 1k Jan 9, 2023
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Facebook Research 296 Dec 29, 2022
(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Active Vision Laboratory 411 Dec 26, 2022
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters

[Unofficial code-base] NeRF--: Neural Radiance Fields Without Known Camera Parameters [ Project | Paper | Official code base ] ⬅️ Thanks the original

Jianfei Guo 239 Dec 22, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF implementation. Contact Jon Barron if you encounter any issues.

Google 625 Dec 30, 2022
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs Check out the paper on arXiv: https://arxiv.org/abs/2103.13744 This repo cont

Christian Reiser 373 Dec 20, 2022
Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

null 111 Dec 29, 2022
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

null 551 Dec 29, 2022
DeRF: Decomposed Radiance Fields

DeRF: Decomposed Radiance Fields Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, Andrea Tagliasacchi Links Paper Project Page Abstract

UBC Computer Vision Group 24 Dec 2, 2022
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Depth-supervised NeRF: Fewer Views and Faster Training for Free Project | Paper | YouTube Pytorch implementation of our method for learning neural rad

null 524 Jan 8, 2023
Unsupervised Discovery of Object Radiance Fields

Unsupervised Discovery of Object Radiance Fields by Hong-Xing Yu, Leonidas J. Guibas and Jiajun Wu from Stanford University. arXiv link: https://arxiv

Hong-Xing Yu 148 Nov 30, 2022
PyTorch implementation for MINE: Continuous-Depth MPI with Neural Radiance Fields

MINE: Continuous-Depth MPI with Neural Radiance Fields Project Page | Video PyTorch implementation for our ICCV 2021 paper. MINE: Towards Continuous D

Zijian Feng 325 Dec 29, 2022
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

Facebook Research 281 Dec 22, 2022
BARF: Bundle-Adjusting Neural Radiance Fields 🤮 (ICCV 2021 oral)

BARF ?? : Bundle-Adjusting Neural Radiance Fields Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey IEEE International Conference on Comp

Chen-Hsuan Lin 539 Dec 28, 2022
[ICCV21] Self-Calibrating Neural Radiance Fields

Self-Calibrating Neural Radiance Fields, ICCV, 2021 Project Page | Paper | Video Author Information Yoonwoo Jeong [Google Scholar] Seokjun Ahn [Google

null 381 Dec 30, 2022
[ICCV 2021 Oral] NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

NerfingMVS Project Page | Paper | Video | Data NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo Yi Wei, Shaohui

Yi Wei 369 Dec 24, 2022
This is the code for "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields".

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields This is the code for "HyperNeRF: A Higher-Dimensional

Google 702 Jan 2, 2023
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

NeRF-pytorch NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are

Yen-Chen Lin 3.2k Jan 8, 2023