This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Overview

Deformable Neural Radiance Fields

This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

This codebase contains a re-implementation of Nerfies using JAX, building on JaxNeRF. We have been careful to match implementation details and have reproduced the original results presented in the paper.

Demo

We provide an easy-to-get-started demo using Google Colab!

These Colabs will allow you to train a basic version of our method using Cloud TPUs (or GPUs) on Google Colab.

Note that due to limited compute resources available, these are not the fully featured models. If you would like to train a fully featured Nerfie, please refer to the instructions below on how to train on your own machine.

Description Link
Process a video into a Nerfie dataset Open In Colab
Train a Nerfie Open In Colab
Render a Nerfie video Open In Colab

Setup

The code can be run under any environment with Python 3.7 and above. (It may run with lower versions, but we have not tested it).

We recommend using Miniconda and setting up an environment:

conda create --name nerfies python=3.8

Next, install the required packages:

pip install -r requirements.txt

Install the appropriate JAX distribution for your environment by following the instructions here. For example:

# For CUDA version 11.0
pip install --upgrade jax jaxlib==0.1.57+cuda110 -f https://storage.googleapis.com/jax-releases/jax_releases.html

# For CUDA version 10.1
pip install --upgrade jax jaxlib==0.1.57+cuda101 -f https://storage.googleapis.com/jax-releases/jax_releases.html

Training

After preparing a dataset, you can train a Nerfie by running:

export DATASET_PATH=/path/to/dataset
export EXPERIMENT_PATH=/path/to/save/experiment/to
python train.py \
    --data_dir $DATASET_PATH \
    --exp_dir $EXPERIMENT_PATH \
    --gin_configs configs/test_vrig.gin

To plot telemetry to Tensorboard and render checkpoints on the fly, also launch an evaluation job by running:

python eval.py \
    --data_dir $DATASET_PATH \
    --exp_dir $EXPERIMENT_PATH \
    --gin_configs configs/test_vrig.gin

The two jobs should use a mutually exclusive set of GPUs. This division allows the training job to run without having to stop for evaluation.

Configuration

  • We use Gin for configuration.
  • We provide a couple preset configurations.
  • Please refer to config.py for documentation on what each configuration does.
  • Preset configs:
    • gpu_vrig_paper.gin: This is the configuration we used to generate the table in the paper. It requires 8 GPUs for training.
    • gpu_fullhd.gin: This is a high-resolution model and will take around 3 days to train on 8 GPUs.
    • gpu_quarterhd.gin: This is a low-resolution model and will take around 14 hours to train on 8 GPUs.
    • test_local.gin: This is a test configuration to see if the code runs. It probably will not result in a good looking result.
    • test_vrig.gin: This is a test configuration to see if the code runs for validation rig captures. It probably will not result in a good looking result.
  • Training on fewer GPUs will require tuning of the batch size and learning rates. We've provided an example configuration for 4 GPUs in gpu_quarterhd_4gpu.gin but we have not tested it, so please only use it as a reference.

Datasets

A dataset is a directory with the following structure:

dataset
    ├── camera
    │   └── ${item_id}.json
    ├── camera-paths
    ├── rgb
    │   ├── ${scale}x
    │   └── └── ${item_id}.png
    ├── metadata.json
    ├── points.npy
    ├── dataset.json
    └── scene.json

At a high level, a dataset is simply the following:

  • A collection of images (e.g., from a video).
  • Camera parameters for each image.

We have a unique identifier for each image which we call item_id, and this is used to match the camera and images. An item_id can be any string, but typically it is some alphanumeric string such as 000054.

camera

  • This directory contains cameras corresponding to each image.
  • We use a camera model identical to the OpenCV camera model, which is also supported by COLMAP.
  • Each camera is a serialized version of the Camera class defined in camera.py and looks like this:
{
  // A 3x3 world-to-camera rotation matrix representing the camera orientation.
  "orientation": [
    [0.9839, -0.0968, 0.1499],
    [-0.0350, -0.9284, -0.3699],
    [0.1749, 0.358, -0.9168]
  ],
  // The 3D position of the camera in world-space.
  "position": [-0.3236, -3.26428, 5.4160],
  // The focal length of the camera.
  "focal_length": 2691,
  // The principle point [u_0, v_0] of the camera.
  "principal_point": [1220, 1652],
  // The skew of the camera.
  "skew": 0.0,
  // The aspect ratio for the camera pixels.
  "pixel_aspect_ratio": 1.0,
  // Parameters for the radial distortion of the camera.
  "radial_distortion": [0.1004, -0.2090, 0.0],
  // Parameters for the tangential distortion of the camera.
  "tangential": [0.001109, -2.5733e-05],
  // The image width and height in pixels.
  "image_size": [2448, 3264]
}

camera-paths

  • This directory contains test-time camera paths which can be used to render videos.
  • Each sub-directory in this path should contain a sequence of JSON files.
  • The naming scheme does not matter, but the cameras will be sorted by their filenames.

rgb

  • This directory contains images at various scales.
  • Each subdirectory should be named ${scale}x where ${scale} is an integer scaling factor. For example, 1x would contain the original images while 4x would contain images a quarter of the size.
  • We assume the images are in PNG format.
  • It is important the scaled images are integer factors of the original to allow the use of area relation when scaling the images to prevent Moiré. A simple way to do this is to simply trim the borders of the image to be divisible by the maximum scale factor you want.

metadata.json

  • This defines the 'metadata' IDs used for embedding lookups.
  • Contains a dictionary of the following format:
{
    "${item_id}": {
        // The embedding ID used to fetch the deformation latent code
        // passed to the deformation field.
        "warp_id": 0,
        // The embedding ID used to fetch the appearance latent code
        // which is passed to the second branch of the template NeRF.
        "appearance_id": 0,
        // For validation rig datasets, we use the camera ID instead
        // of the appearance ID. For example, this would be '0' for the
        // left camera and '1' for the right camera. This can potentially
        // also be used for multi-view setups as well.
        "camera_id": 0
    },
    ...
},

scene.json

  • Contains information about how we will parse the scene.
  • See comments inline.
{
  // The scale factor we will apply to the pointcloud and cameras. This is
  // important since it controls what scale is used when computing the positional
  // encoding.
  "scale": 0.0387243672920458,
  // Defines the origin of the scene. The scene will be translated such that
  // this point becomes the origin. Defined in unscaled coordinates.
  "center": [
    1.1770838526103944e-08,
    -2.58235339289195,
    -1.29117656263135
  ],
  // The distance of the near plane from the camera center in scaled coordinates.
  "near": 0.02057418950149491,
  // The distance of the far plane from the camera center in scaled coordinates.
  "far": 0.8261601717667288
}

dataset.json

  • Defines the training/validation split of the dataset.
  • See inline comments:
{
  // The total number of images in the dataset.
  "count": 114,
  // The total number of training images (exemplars) in the dataset.
  "num_exemplars": 57,
  // A list containins all item IDs in the dataset.
  "ids": [...],
  // A list containing all training item IDs in the dataset.
  "train_ids": [...],
  // A list containing all validation item IDs in the dataset.
  // This should be mutually exclusive with `train_ids`.
  "val_ids": [...],
}

points.npy

  • A numpy file containing a single array of size (N,3) containing the background points.
  • This is required if you want to use the background regularization loss.

Citing

If you find our work useful, please consider citing:

@article{park2020nerfies
  author    = {Park, Keunhong 
               and Sinha, Utkarsh 
               and Barron, Jonathan T. 
               and Bouaziz, Sofien 
               and Goldman, Dan B 
               and Seitz, Steven M. 
               and Martin-Brualla, Ricardo},
  title     = {Deformable Neural Radiance Fields},
  journal   = {arXiv preprint arXiv:2011.12948},
  year      = {2020},
}
Comments
  • Nerfies_Training.ipynb collab fails on import configs,  having changed no settings

    Nerfies_Training.ipynb collab fails on import configs, having changed no settings

    I ran both collabs in sequence Nerfies_Capture_Processing.ipynb and Nerfies_Training.ipynb on the process of running the second one - having changed nothing igot an error thanks


    ValueError Traceback (most recent call last) in () 6 from IPython.display import display, Markdown 7 ----> 8 from nerfies import configs 9 10

    5 frames /usr/lib/python3.7/dataclasses.py in _get_field(cls, a_name, a_type) 731 # For real fields, disallow mutable defaults for known types. 732 if f._field_type is _FIELD and isinstance(f.default, (list, dict, set)): --> 733 raise ValueError(f'mutable default {type(f.default)} for field ' 734 f'{f.name} is not allowed: use default_factory') 735

    opened by xvdp 7
  • Questions about face processing section in Colab

    Questions about face processing section in Colab

    Hi, thanks for releasing the codes.

    In Colab, process a video into a Nerfie dataset, it seems that the face processing section is optional. I skipped this section and ran compute scene information section. It gives errors, "name 'new_scene_manager' is not defined' ". 'new_scene_manager' is defined in face processing section. Is it necessary to run the face processing section or did I miss anything?

    WeChat Image_20210204220139

    Thank you!

    opened by jiejie22997 6
  • how to use on video that does not contain a face?

    how to use on video that does not contain a face?

    Hi, Currently the pre processing notebook for dataset generation is limited to being used with videos that contain faces despite this being marked as an "optional" step.

    the mediapipe library is used to get a mesh of the face and parts of this are used to generate the scene.json as well as the test camera path.

    Do you have a more generic workflow or simplified method for generating the scene.json?

    opened by kurtjcu 5
  • Flag --base_folder must have a value other than None.

    Flag --base_folder must have a value other than None.

    I encountered this issue when I ran python train.py --data_dir $DATASET_PATH --base_folder $EXPERIMENT_PATH --gin_configs configs/test_vrig.gin. Below is the error log. What can I try to solve this? Thanks!

    Traceback (most recent call last):
      File "train.py", line 54, in <module>
        jax.config.parse_flags_with_absl()
      File "/home/jllantero/miniconda3/envs/nerfies/lib/python3.8/site-packages/jax/_src/config.py", line 161, in parse_flags_with_absl
        absl.flags.FLAGS(jax_argv, known_only=True)
      File "/home/jllantero/miniconda3/envs/nerfies/lib/python3.8/site-packages/absl/flags/_flagvalues.py", line 673, in __call__
        self.validate_all_flags()
      File "/home/jllantero/miniconda3/envs/nerfies/lib/python3.8/site-packages/absl/flags/_flagvalues.py", line 533, in validate_all_flags
        self._assert_validators(all_validators)
      File "/home/jllantero/miniconda3/envs/nerfies/lib/python3.8/site-packages/absl/flags/_flagvalues.py", line 568, in _assert_validators
        raise _exceptions.IllegalFlagValueError('\n'.join(messages))
    absl.flags._exceptions.IllegalFlagValueError: flag --base_folder=None: Flag --base_folder must have a value other than None.
    
    opened by joiellantero 4
  • Rendering results are strange

    Rendering results are strange

    In training process, predicted RGB of validation dataset is fine. 2021-05-07 10-03-44 的屏幕截图 But when rendering with the latest checkpoint, I got the strange results as follows: 2021-05-07 10-06-18 的屏幕截图

    opened by RodneyPerhaps 4
  • Error with Nerfies configuration in

    Error with Nerfies configuration in "Nerfies Render Video.ipynb".

    Hi, I have been using your Google Colab notebooks for several days. They were working fine but yesterday I tried to use "Nerfies Render Video.ipynb" and I got this error:

    image

    I have tried changing the "train_dir:" and "data_dir:", but I still get the same error. Any ideas on how to solve this problem?

    opened by SaukVA 4
  • JAX error: RuntimeError: optional has no value

    JAX error: RuntimeError: optional has no value

    When Parse data in https://github.com/google/nerfies/blob/main/notebooks/Nerfies_Capture_Processing.ipynb

    executing following cell:

    if colmap_image_scale > 1:
      print(f'Scaling COLMAP cameras back to 1x from {colmap_image_scale}x.')
      for item_id in scene_manager.image_ids:
        camera = scene_manager.camera_dict[item_id]
        scene_manager.camera_dict[item_id] = camera.scale(colmap_image_scale)
    

    Run into error below, seems to be a JAX error, how should we address this? Thanks

    Scaling COLMAP cameras back to 1x from 4x.
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-24-32db5f4d2f95> in <module>
         11   for item_id in scene_manager.image_ids:
         12     camera = scene_manager.camera_dict[item_id]
    ---> 13     scene_manager.camera_dict[item_id] = camera.scale(colmap_image_scale)
         14 
         15 
    
    /usr/local/lib/python3.8/dist-packages/nerfies/camera.py in scale(self, scale)
        315         radial_distortion=self.radial_distortion.copy(),
        316         tangential_distortion=self.tangential_distortion.copy(),
    --> 317         image_size=jnp.array((int(round(self.image_size[0] * scale)),
        318                               int(round(self.image_size[1] * scale)))),
        319     )
    
    /usr/local/lib/python3.8/dist-packages/jax/_src/numpy/lax_numpy.py in array(object, dtype, copy, order, ndmin)
       2903     _inferred_dtype = object.dtype and dtypes.canonicalize_dtype(object.dtype)
       2904     lax._check_user_dtype_supported(_inferred_dtype, "array")
    -> 2905     out = _device_put_raw(object, weak_type=weak_type)
       2906     if dtype: assert _dtype(out) == dtype
       2907   elif isinstance(object, (DeviceArray, core.Tracer)):
    
    /usr/local/lib/python3.8/dist-packages/jax/_src/lax/lax.py in _device_put_raw(x, weak_type)
       1493   else:
       1494     aval = raise_to_shaped(core.get_aval(x), weak_type=weak_type)
    -> 1495     return xla.array_result_handler(None, aval)(*xla.device_put(x))
       1496 
       1497 def iota(dtype: DType, size: int) -> Array:
    
    /usr/local/lib/python3.8/dist-packages/jax/interpreters/xla.py in make_device_array(aval, device, device_buffer)
       1032   if (isinstance(device_buffer, _CppDeviceArray)):
       1033 
    -> 1034     if device_buffer.aval == aval and device_buffer._device == device:
       1035       return device_buffer
       1036     device_buffer = device_buffer.clone()
    
    RuntimeError: optional has no value
    
    opened by mengdong 4
  • TypeError in rendering video frames.

    TypeError in rendering video frames.

    Hi, Thanks a lot for releasing the code! I tried running the Colab demo, but got errors (as shown in figure) when rendering a video both in training and rendering notebook. How to solve this problem? (Is it related to the WARNING at the top of the figure? And how to set the 'base_64' param mentioned in the warning?) fig

    opened by zoe2718 4
  • How to get points.npy

    How to get points.npy

    I noticed that points.npy is included in the dataset for background regularization. But it can't be created in the demo of "Process a video into a Nerfie dataset". So how can I get it?

    opened by longtimenotalk2 3
  • can not import flax in self hostet jupyter notebook

    can not import flax in self hostet jupyter notebook

    Hi, thanks to anyone involved publishing this. I managed to get the video extraction notebook running after installing heaps of packages missing from the 'requirements.txt'.

    Now I'm stuck in the second notebook, trying to import flax. Running the following in the same cell yields:

    !python -c "import flax; print(flax)"
    import flax

    <module 'flax' from '/opt/conda/envs/nerfies/lib/python3.8/site-packages/flax/init.py'>

    ModuleNotFoundError Traceback (most recent call last) in 1 get_ipython().system('python -c "import flax; print(flax)"') ----> 2 import flax ModuleNotFoundError: No module named 'flax'

    Does anyone reading this have a clue what's up? Cheers, Magnus

    opened by sungam94 3
  • Unable to run on GPU

    Unable to run on GPU

    I am trying to run Nerfies on GPU it always "RuntimeError: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16504591536 bytes."

    I tried it locally and on colab (selected GPU runtime and GPU in code)

    Both places same error.

    It happens at cell "Train a Nerfie!" on line "with time_tracker.record_time('train_step'): state, stats, keys = ptrain_step(keys, state, batch, scalar_params) time_tracker.toc('total')"

    Any solution to this??

    opened by Ntweat 2
  • Results shifted to some side

    Results shifted to some side

    Hello,

    thank you for this work first of all. Currently I'm trying to reimplement this work in Pytorch and I'm facing a specific problem.

    My problem is, that the rendered images seem to be shifted to a wrong direction, if I have to describe it like someone moved the camera. That's why I wanted to ask if someone could have an idea where I did my mistake.

    I tried to follow this implementation of Nerfies as close as possible but written in Pytorch.

    Here I have two example pictures, with the left being the rendered one and the right being the ground truth. I'm thankful for any help here.

    media_images_Validation-Images_1625_9c95e806c92a981a2708

    media_images_Validation-Images_1624_198dcd6cb682420d7088

    opened by kemoyin 0
  • connot gin-config @ git+https://github.com/google/gin-config@243ba87b3fcfeb2efb4a920b8f19679b61a6f0dc

    connot gin-config @ git+https://github.com/google/gin-config@243ba87b3fcfeb2efb4a920b8f19679b61a6f0dc

    When I use pip pip install -r requirements.txt,I found I can not link to https://github.com/google/gin-config@243ba87b3fcfeb2efb4a920b8f19679b61a6f0dc. And the error is: fatal: Couldn't find remote ref 243ba87b3fcfeb2efb4a920b8f19679b61a6f0dc, so how can i solve this,thanks.

    opened by Hansen7777777 0
  • Project dependencies may have API risk issues

    Project dependencies may have API risk issues

    Hi, In nerfies, inappropriate dependency versioning constraints can cause risks.

    Below are the dependencies and version constraints that the project is using

    absl-py==0.13.0
    flax==0.3.4
    gin-config@git+https://github.com/google/gin-config@243ba87b3fcfeb2efb4a920b8f19679b61a6f0dc
    imageio==2.9.0
    immutabledict==2.2.0
    jax==0.2.20
    numpy==1.19.5
    opencv-python==4.5.3.56
    Pillow==8.3.2
    scikit-image==0.18.3
    scipy==1.7.1
    tensorboard==2.6.0
    tensorflow>=2.6.1
    

    The version constraint == will introduce the risk of dependency conflicts because the scope of dependencies is too strict. The version constraint No Upper Bound and * will introduce the risk of the missing API Error because the latest version of the dependencies may remove some APIs.

    After further analysis, in this project, The version constraint of dependency absl-py can be changed to >=0.3.0,<=1.1.0. The version constraint of dependency flax can be changed to >=0.2.1,<=0.5.3. The version constraint of dependency gin-config can be changed to ==0.5.0. The version constraint of dependency numpy can be changed to >=1.8.0,<=1.23.0rc3. The version constraint of dependency Pillow can be changed to ==9.2.0. The version constraint of dependency Pillow can be changed to >=2.0.0,<=9.1.1. The version constraint of dependency scipy can be changed to >=1.2.0rc1,<=1.5.4.

    The above modification suggestions can reduce the dependency conflicts as much as possible, and introduce the latest version as much as possible without calling Error in the projects.

    The invocation of the current project includes all the following methods.

    The calling methods from the absl-py
    absl.logging.log_every_n_seconds
    
    The calling methods from the flax
    flax.training.checkpoints.restore_checkpoint
    flax.linen.Embed
    flax.linen.vmap
    flax.linen.Dense
    flax.training.checkpoints.save_checkpoint
    flax.optim.Adam
    flax.linen.sigmoid
    
    The calling methods from the gin-config
    gin.config.markdown
    
    The calling methods from the numpy
    numpy.linalg.norm
    
    The calling methods from the Pillow
    PIL.Image.fromarray
    
    The calling methods from the scipy
    scipy.spatial.transform.Rotation.from_dcm
    scipy.interpolate.interp1d
    scipy.spatial.transform.Slerp
    
    The calling methods from the all methods
    jax.numpy.argmax
    jax.tree_multimap.items
    self.point3D_id_to_point3D_idx.items
    MLP
    self.output_activation
    gin.config.external_configurable
    set.discard
    cv2.CVX_64F.gray.cv2.Laplacian.var
    v_warp_field_cls
    self.appearance_encoder
    load_image
    nerfies.image_utils.rescale_image
    self.rng.permutation
    nerfies.models.construct_nerf
    os.path.join
    point3D_id.self.point3D_id_to_images.tobytes
    jax.numpy.stack
    setuptools.setup
    process_iterator
    self.images.iterkeys
    super.__init__
    jax.numpy.trace
    im
    nerfies.image_utils.save_image
    points.reshape.reshape
    item_ids.self.rng.permutation.tolist
    FileExistsError
    _compute_loss_and_stats
    jax.numpy.clip
    rgb_mlp
    nerfies.utils.TimeTracker
    numpy.any
    numpy.asarray.min
    c.reshape.reshape
    point3D_idx.self.point3D_errors.tobytes
    numpy.squeeze
    jac_fn
    self.glo_encoder
    self.exists
    tensorflow.io.gfile.makedirs
    collections.defaultdict
    AnnealedSinusoidalEncoder
    jax.nn.initializers.glorot_uniform
    jax.jacfwd
    from_config
    nerfies.gpath.GPath.exists
    f.readline.strip
    flax.metrics.tensorboard.SummaryWriter.image
    self._create_preloaded_dataset
    d.pop.SCHEDULE_MAP
    tensorflow.io.gfile.GFile
    join
    tensorflow.io.gfile.listdir
    q.q.copy
    flax.optim.Adam.create
    jax.numpy.min
    self.radial_distortion.copy
    numpy.uint16.a.astype.clip.astype
    super
    camera.get_camera_matrix
    SinusoidalEncoder
    scipy.interpolate.interp1d
    path.suffix.lstrip
    sorted
    model.apply
    state.replace
    dataset.shuffle.map
    self.load_points3D
    nerfies.rigid_body.to_homogenous
    DualQuaternion
    compute_opaqueness_mask.astype
    expm1_safe
    nerfies.gpath.GPath.open
    tensorflow.shape
    self.save_points3D
    optimizer.apply_gradient
    positions.append
    self.embed
    cs
    absl.app.run
    jax.numpy.block
    jax.tree_util.tree_map
    Camera
    self.get_parameters.items
    numpy.cos
    v.mean
    flax.jax_utils.prefetch_to_device
    process_batch
    self.posenc
    dense
    COLMAPDatabase.connect.add_matches
    tensorflow.broadcast_to
    self.load_camera.scale
    iterator_from_dataset
    jax.numpy.zeros_like
    cond_2
    jax.value_and_grad
    summary_dir.mkdir
    jax.vmap
    collections.OrderedDict
    jax.device_get
    numpy.mean
    nerfies.utils.tree_collate.items
    jax.lax.pmean
    checkpoint_dir.mkdir
    flax.metrics.tensorboard.SummaryWriter.scalar
    scipy.optimize.root
    absl.logging.warning
    self.image_path.endswith
    jax.random.split
    rgb.rgb_target.sum
    nerfies.datasets.iterator_from_dataset
    self.get_camera_matrix
    jax.numpy.diag
    nerfies.utils.strided_subset
    nerfies.rigid_body.exp_se3
    self.point3D_id_to_point3D_idx.get
    jax.numpy.broadcast_to
    image.q.q.tobytes
    self.update
    isinstance
    executor.map
    array_to_string
    self.distortion_func
    nerfies.datasets.from_config.create_cameras_dataset
    self.warp_field
    rp_to_se3
    len
    flax.jax_utils.unreplicate
    numpy.atleast_2d.dot
    tensorflow.io.gfile.glob
    flax.jax_utils.replicate
    compute_depth_map
    tuple
    dataset.shuffle.repeat
    scipy.spatial.transform.Rotation.from_dcm
    height.scale_factor.is_integer
    base
    nerfies.gpath.GPath
    nerfies.utils.shard
    numpy.random.RandomState
    self.appearance_ids.index
    jax.numpy.expand_dims
    numpy.uint32.data.np.array.reshape
    cross_prod_matrix
    jax.numpy.cos
    extra.get
    jax.random.normal
    tr_positive
    jax.numpy.exp
    self.cameras.items
    RuntimeError
    f.read.iter.c.c.join.decode
    self._meters.items
    jax.lax.stop_gradient
    nerfies.modules.NerfMLP
    tensorflow.python.data.util.nest.flatten_up_to
    nerfies.datasets.from_config.load_points
    jax.numpy.greater_equal
    self._save_cameras_txt
    self.metadata_encoder
    flax.linen.initializers.xavier_uniform
    jax.numpy.dot
    cross_prod_matrix.dot
    jax.numpy.searchsorted
    list
    metadata_path.exists
    tensorflow.linalg.matvec
    nerfies.datasets.nerfies.NerfiesDataSource
    self.points_encoder
    nerfies.configs.EvalConfig
    value.reshape
    time.time
    self.encode_metadata
    self.summary
    renders_dir.mkdir
    jax.numpy.pad
    enumerate
    self.load_camera.get_pixel_centers
    tensorflow.experimental.numpy.zeros_like
    float
    os.path.exists
    trunk_mlp
    jax.numpy.eye
    checkpoint_dir.exists
    nerfies.utils.TimeTracker.summary
    self.get_item
    model.create_warp_field.apply
    self.q0.q.dot
    absl.flags.DEFINE_multi_string
    numpy.sqrt
    list.discard
    hasattr
    format
    jax.numpy.log
    numpy.random.seed
    PIL.Image.fromarray.astype
    camera.pixels_to_rays.astype
    image.Image
    jax.numpy.full
    PIL.Image.fromarray.save
    flax.metrics.tensorboard.SummaryWriter
    delete_old_renders
    self.copy
    learning_rate_sched
    min
    numpy.full
    depth_exp.depth_target.abs.squeeze
    self.q0.norm
    tensorflow.cast
    matplotlib.colors.LinearSegmentedColormap.from_list
    ValueError
    self.point3D_id_to_point3D_idx.values
    out_cameras.append
    jax.process_count
    dataset.shuffle.batch
    PIL.Image.fromarray
    x.shape.xu.reshape.self.self.distortion_func.x.ravel
    tensorflow.range
    numpy.tile
    camera_dir.exists
    numpy.square
    self.tangential_distortion.copy
    nerfies.camera.Camera.from_json
    collections.defaultdict.items
    image_to_float32
    _build_colormap
    line.strip
    copy.copy.pop
    fh.read
    base_encoder
    self.ToR
    nerfies.model_utils.vmap_module
    self.load_images
    self.norm
    jax.numpy.mean
    jax.numpy.cumsum
    numpy.empty
    nerfies.tf_camera.TFCamera
    reshape_image
    Quaternion.FromR
    self.q.ToR
    nerfies.utils.jacobian_to_curl
    self.cosine_easing_window
    nerfies.utils.TimeTracker.tic
    jax.config.parse_flags_with_absl
    dict.pop
    writer.histogram
    nerfies.datasets.from_config.load_test_cameras
    os.makedirs
    IOError
    nerfies.model_utils.compute_depth_index
    self.get_warp_id
    Quaternion
    jax.numpy.sort
    nerfies.image_utils.save_depth
    downsample_image
    iter
    nerfies.schedules.from_config
    tensorflow.python.data.util.nest.flatten
    gin.config.markdown
    self.GetNameFromType
    x.objective.root.x.reshape.reshape
    model.create_warp_field
    jax.numpy.divide
    COLMAPDatabase.connect.add_keypoints
    meter.reduce
    COLMAPDatabase.connect.add_camera
    numpy.zeros
    f.read
    out_times.slerp.as_dcm
    numpy.where
    tensorflow.experimental.numpy.stack
    point3D_idx.self.points3D.tobytes
    nerfies.model_utils.TrainState
    complex
    conjugate
    yd.copy
    self.executescript
    numpy.rec.fromarrays
    jax.nn.softplus
    numpy.kron
    from_tuple
    jax.numpy.clip.reshape
    nerfies.utils.unshard
    set
    self.warp_ids.index
    _load_dataset_ids
    numpy.random.randint
    jax.numpy.split
    tensorflow.experimental.numpy.where
    flax.linen.Embed
    UINT8_MAX.UINT8_MAX.image.clip.astype
    cond_idx
    flax.linen.vmap
    rays_dir.reshape.reshape
    jax.numpy.eye.jtj.sum
    struct.Struct
    UINT16_MAX.image.clip
    nerfies.utils.TimeTracker.record_time
    nerfies.model_utils.sample_along_rays
    self._save_cameras_bin
    jax.numpy.where
    state.replace.replace
    numpy.asarray.copy
    range
    copy.copy
    ret_maps.append
    jax.random.choice
    jax.numpy.array
    numpy.array
    numpy.asarray.max
    self.base_schedule
    self.trunk
    tensorflow.io.gfile.rmtree
    self.get_time_id
    numpy.ascontiguousarray
    metadata_path.open
    nerfies.configs.ExperimentConfig
    numpy.cross
    self._load_images_txt
    abs
    rotations.append
    self.load_camera
    numpy.linalg.norm
    jax.local_devices
    numpy.expand_dims
    scalar_params.replace.replace
    nerfies.visualization.colorize
    image.tvec.tobytes
    imageio.imread
    NerfModel.init
    any
    rgb.weights.sum
    numpy.argmax
    self._load_points3D_txt
    tqdm.tqdm
    render_fn
    nerfies.glo.GloEncoder
    math.ceil
    flax.linen.initializers.uniform
    jax.numpy.concatenate
    alpha_mlp
    log1p_safe
    dataset.shuffle.unbatch
    tensorflow.io.gfile.mkdir
    ref_camera.copy
    jax.numpy.sqrt
    key.self._meters.update
    issubclass
    self.R
    cv2.Laplacian
    x.reshape
    nerfies.training.save_checkpoint
    tensorflow.TensorSpec
    qsq.sum
    v_elastic_fn
    numpy.concatenate._numpy
    tensorflow.experimental.numpy.abs
    nerfies.configs.TrainConfig
    colormap
    numpy.zeros_like
    camera.Camera
    tensorflow.experimental.numpy.array
    numpy.uint64
    self.points3D.append
    numpy.std
    jax.tree_multimap
    json.load
    flax.linen.Dense
    jax.tree_util.tree_multimap
    self._get_item_signature
    self.get_appearance_id
    self.camera_encoder
    line.startswith
    nerfies.gpath.GPath.glob
    numpy.atleast_2d
    self.folder.endswith
    rotation.Quaternion
    cv2.imdecode
    jax.jit
    colorize_depth
    self.save_images
    numpy.allclose
    sqlite3.connect
    self.principal_point.copy
    dict
    self._create_lazy_dataset
    jax.numpy.squeeze
    self.mlp
    absl.logging.info
    xs.interp.astype
    numpy.stack.reshape
    self.point3D_ids.append
    cond_1
    self.R.T.dot
    width.scale_factor.is_integer
    jax.numpy.expm1
    self._save_images_txt
    COLMAPDatabase.connect
    normalize
    immutabledict.immutabledict
    numpy.eye
    jax.numpy.linspace
    i.decode
    numpy.take
    background_loss.mean.mean
    self.make_rng
    self.point3D_id_to_images.items
    GPath
    nerfies.modules.AnnealedSinusoidalEncoder
    self.load_depth
    rgb_mlp.reshape
    self.sigma_activation
    numpy.sin
    load_scene_info
    safe_norm
    flax.training.checkpoints.restore_checkpoint
    next
    self.trunk_width.dense
    f.write
    self.__mul__
    _log_to_tensorboard
    image.point3D_ids.astype
    _radial_and_tangential_undistort
    reduction.self.summary.items
    nearest_rotation_svd
    absl.flags.mark_flag_as_required
    alpha_mlp.reshape
    warp_alpha_sched
    self.COLMAPDatabase.super.__init__
    exit
    dtype.blob.np.frombuffer.reshape
    inv_scale.is_integer
    nerfies.image_utils.image_to_uint8
    get_colormap
    self.data_dir.open
    self.load_camera.pixels_to_rays
    numpy.max
    jax.numpy.sin
    nerfies.model_utils.volumetric_rendering
    re
    nerfies.datasets.core.load_camera
    absl.flags.DEFINE_string
    renders_dir.exists
    zip
    _camera_to_rays_fn
    jax.numpy.maximum
    spline
    camera.get_params.tobytes
    multiply
    tensorflow.stack
    flax.linen.sigmoid
    numpy.arctan2
    self._meters.values
    self.pixels_to_rays
    jax.local_device_count
    nerfies.datasets.from_config.create_iterator
    tensorflow.data.Dataset.from_generator
    camera.get_params
    R.Quaternion.FromR.ToAxisAngle
    jax.numpy.linalg.norm
    numpy.getbuffer
    jax.device_count
    elastic_loss_weight_sched
    elastic_loss.sum.mean
    save_image
    struct.Struct.pack
    array_to_blob
    tensorflow.image.ssim_multiscale
    self.load_camera.get_parameters
    u.dot
    branch.stats.items
    self._load_cameras_txt
    v.reduce.values
    struct.unpack
    nerfies.rigid_body.from_homogenous
    jax.numpy.linalg.svd
    jax.numpy.ones
    k.meters.update
    item_id.replace.replace
    jax.numpy.take_along_axis
    stats.items
    bytearray
    nerfies.training.ScalarParams
    jax.numpy.reshape.flatten
    point3D_idx.self.point3D_colors.tobytes
    jax.pmap
    flax.optim.Adam
    path.rmtree
    jax.numpy.reshape
    self.load_cameras
    cond_3
    Exception
    numpy.floor
    round
    time_dict.items
    nerfies.utils.TimeTracker.toc
    tensorflow.convert_to_tensor
    minmax
    str
    self.camera_ids.index
    jax.process_indexs
    camera.get_parameters.get
    _compute_residual_and_jacobian
    numpy.clip
    v.reduce
    scale_values
    jax.numpy.tile
    self._values.clear
    inv_softplus
    out.update
    numpy.tile.astype
    jax.numpy.nanmean
    gin.parse_config_files_and_bindings
    numpy.square.sum
    jax.numpy.ones_like
    norm
    _load_image
    line.split
    item.pop
    self._save_points3D_bin
    self.get_parameters
    numpy.arange
    numpy.concatenate
    interp
    z_vals.weights.sum
    self.load_colmap_project_file
    self.points_to_local_points
    jax.numpy.max
    compute_multiscale_ssim
    set.isdisjoint
    logit
    elastic_loss.sum.mean.sum
    nerfies.datasets.from_config
    float.is_integer
    numpy.uint16.a.astype.clip
    copy.deepcopy
    time_alpha_sched
    upsample_image
    numpy.broadcast_to
    jax.process_index
    tensorflow.python.data.util.nest.pack_sequence_as
    tensorflow.numpy_function
    self._load_cameras_bin
    _norm
    camera.get_pixel_centers.astype
    numpy.uint16.a.astype.clip.clip
    self.create_warp_field
    self.hidden_activation
    get_image_ids_from_pair_id
    jax.numpy.power
    exp_so3
    flax.training.checkpoints.save_checkpoint
    scipy.interpolate.CubicSpline
    self.save_cameras
    level.self.nerf_mlps
    jax.numpy.sum
    safe_acos
    functools.partial
    weights.sum
    numpy.asarray.ravel
    map
    layer
    jax.numpy.nanmedian
    tensorflow.io.gfile.isdir
    jax.numpy.cumprod
    NotImplementedError
    v.tolist
    max
    points.astype.astype
    image.np.asarray.astype
    numpy.linspace
    numpy.ones_like
    numpy.asarray
    nerfies.modules.MLP
    self.q.dot
    tensorflow.meshgrid
    self.getT
    camera.Camera.GetNumParams
    jax.numpy.cross
    self.point3D_errors.append
    numpy.load
    numpy.arcsin
    camera_dir.glob
    nerfies.camera.Camera
    numpy.min
    numpy.random.rand
    numpy.stack
    nerfies.model_utils.noise_regularize
    process_batch.items
    nerfies.utils.tree_collate
    summary_dir.exists
    jax.numpy.log1p
    COLMAPDatabase.connect.add_image
    tensorflow.experimental.numpy.sum
    rgb.rgb_target.abs.sum
    interpolate_colormap
    int
    nerfies.configs.ModelConfig
    setuptools.find_packages
    self.load_rgb
    jax.numpy.arccos
    jax.numpy.linalg.det
    numpy.trace
    blob_to_array
    tensorflow.data.Dataset.from_tensor_slices
    self.render_samples
    nerfies.gpath.GPath.mkdir
    jax.tree_map
    jax.numpy.logical_xor
    tensorflow.experimental.numpy.sqrt
    numpy.identity
    compute_background_loss
    self.point3D_valid
    tensorflow.dtypes.as_dtype
    image.q.ToR
    tensorflow.io.gfile.exists
    nerfies.utils.jacobian_to_div
    nerfies.modules.SinusoidalEncoder.model_utils.vmap_module
    tensorflow.experimental.numpy.ones_like
    self.position.copy
    self.get_condition_inputs
    cls
    numpy.prod
    jax.devices
    numpy.float64
    f.readline
    numpy.radians
    self.get_time
    gin.configurable
    self.branches
    losses.values
    self._load_images_bin
    jax.numpy.zeros
    ptrain_step
    os.remove
    absl.logging.log_every_n_seconds
    matplotlib.cm.get_cmap
    meter.reset
    self.get_camera_id
    self.images.items
    nerfies.utils.parallel_map
    skew
    grad_fn
    jax.lax.all_gather
    compute_opaqueness_mask
    self.get_inverse_camera_matrix
    from_dict
    nerfies.utils.TimeTracker.reset
    numpy.rec.fromarrays.tobytes
    time.sleep
    COLMAPDatabase.connect.initialize_tables
    math.cos
    self.warp
    numpy.concatenate.reshape
    numpy.uint16.b.astype.clip
    image_to_uint16
    points3D.dot.dot
    itertools.combinations
    piecewise_constant_pdf
    self.time_encoder
    writer.scalar
    self.point3D_colors.append
    type
    self.viewdir_encoder
    self.q0.dot
    depth_med.depth_target.abs.squeeze
    logit_layer
    jax.nn.sigmoid
    jax.numpy.finfo
    nerfies.modules.TimeEncoder
    render_dir.iterdir
    jax.numpy.concatenate.append
    xd.copy
    gin.operative_config_str
    self.images.values
    self._save_points3D_txt
    self.get
    concurrent.futures.ThreadPoolExecutor
    jax.numpy.abs
    nerfies.model_utils.sample_pdf
    jax.random.uniform
    numpy.abs
    x.np.square.sum
    nerfies.utils.general_loss_with_squared_residual
    nerfies.utils.TimeTracker.summary_str
    nerfies.warping.create_warp_field
    broadcast_condition
    data.np.uint32.map.list.np.array.reshape
    self.create_dataset
    itertools.chain
    batch.rgb.mean
    numpy.ones
    self._save_images_bin
    jax.numpy.asarray
    fid.seek
    numpy.all
    xs.cs.astype
    self._values.append
    self.point_encoder
    ret_t.set_shape
    jax.numpy.minimum
    jax.random.PRNGKey
    functools.lru_cache
    jax.numpy.full_like
    print
    scalar_params.warp_reg_loss_scale.scalar_params.warp_reg_loss_alpha.warp_reg_residual.utils.general_loss_with_squared_residual.mean
    numpy.matmul
    cv2.resize
    numpy.uint16.b.astype.clip.astype
    numpy.reshape
    numpy.savetxt
    UINT16_MAX.UINT16_MAX.image.clip.astype
    numpy.finfo
    struct.pack
    numpy.frombuffer
    UINT8_MAX.image.clip
    model_fn
    dataset.shuffle.shuffle
    cv2.cvtColor
    open
    q0.dot
    self._load_points3D_bin
    self.load_camera.copy
    COLMAPDatabase.connect.close
    self.pixel_to_local_rays
    _log_histograms
    sum
    COLMAPDatabase.connect.execute
    tensorflow.config.experimental.set_visible_devices
    model_out.model_out.sum
    numpy.meshgrid
    x.objective.root.x.reshape
    fid.write
    slerp
    batch.model_out.mean
    absl.flags.DEFINE_enum
    scipy.spatial.transform.Slerp
    flax.metrics.tensorboard.SummaryWriter.text
    get_pair_id
    NerfModel
    self.orientation.copy
    save_dir.mkdir
    nerfies.utils.compute_psnr
    schedule.get
    

    @developer Could please help me check this issue? May I pull a request to fix it? Thank you very much.

    opened by PyDeps 0
  • How to do register stereo sequences captured by multi-view rig with COLMAP?

    How to do register stereo sequences captured by multi-view rig with COLMAP?

    Hello,

    first of all, thank you for releasing the amazing work that is Nerfies. I was wondering what code / method was used to register the stereo RGB sequences captured by the multi-view camera rig in the quantitative evaluation portion of the paper.

    It was mentioned in page 7 of the paper that "We register the images using COLMAP [48] with rigid relative camera pose constraints," but I couldn't find the piece of code in the nerfies codebase that carries out this left-right camera registration. I need this code to register my custom dataset captured by a multi-view camera rig.

    I would really appreciate your help on this matter. Best, Andrew.

    opened by andrewsonga 0
  • Colab GPU error: Could not connect to any X display

    Colab GPU error: Could not connect to any X display

    I have Colab Pro and a total of 80 (4k res) frames that I have been trying to process with the Process a video into a Nerfie dataset Colab notebook but facing the following error (please see the screenshot below) while enabling the GPU runtime for Colmap feature matching. I have faced the same error with the feature extraction --SiftExtraction.use_gpu flag too but since feature extraction is done only once for each image, I managed to run it with CPU alone. However, it is not working out with the matching process - it is extremely slow and often gets hung at the first image.

    Can you please help me with this? The feature matching has become a bottleneck for my entire process given the resolution of the frames. Per Colmap docs, it is more efficient to use GPU for extraction, matching, and bundle adjustment and I would like to employ them for the three processes. Thank you in advance.

    To be precise, this is the error as can be seen at the bottom in the screenshot: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root' qt.qpa.screen: QXcbConnection: Could not connect to display Could not connect to any X display.

    Screenshot from 2022-09-28 18-58-25

    opened by sameersharma84 0
Releases(0.1)
  • 0.1(Jun 18, 2021)

    This is a release of the validation rig datasets used in the evaluation of our paper. Please refer to the documentation in the README of the project.

    Notes:

    • We use a pinhole camera model with tangential and radial distortion, identical to the OpenCV model as described in the README. You must take into account the distortion parameters or else the rays will not be correct. This is easy if you use our camera class.
    • As mentioned in the paper, the "Curls" dataset is captured using a synchronized capture app while the rest are captured by manually synchronizing two separately captured videos using audio cues. This means that the exposure will be identical for the former but not for the latter.
    • The train/test split is given in the dataset.json file of each dataset.
    • For our quantitative experiments, we used scale 4x.
    • If you are not using our dataloader, make sure you scale the cameras correctly to match the corresponding scales.
    Source code(tar.gz)
    Source code(zip)
    nerfies-vrig-dataset-v0.1.zip(1777.21 MB)
Owner
Google
Google ❤️ Open Source
Google
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Facebook Research 296 Dec 29, 2022
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs Check out the paper on arXiv: https://arxiv.org/abs/2103.13744 This repo cont

Christian Reiser 373 Dec 20, 2022
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Depth-supervised NeRF: Fewer Views and Faster Training for Free Project | Paper | YouTube Pytorch implementation of our method for learning neural rad

null 524 Jan 8, 2023
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

Facebook Research 281 Dec 22, 2022
This is the code for "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields".

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields This is the code for "HyperNeRF: A Higher-Dimensional

Google 702 Jan 2, 2023
Code release for NeRF (Neural Radiance Fields)

NeRF: Neural Radiance Fields Project Page | Video | Paper | Data Tensorflow implementation of optimizing a neural representation for a single scene an

null 6.5k Jan 1, 2023
(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Active Vision Laboratory 411 Dec 26, 2022
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters

[Unofficial code-base] NeRF--: Neural Radiance Fields Without Known Camera Parameters [ Project | Paper | Official code base ] ⬅️ Thanks the original

Jianfei Guo 239 Dec 22, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF implementation. Contact Jon Barron if you encounter any issues.

Google 625 Dec 30, 2022
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

null 551 Dec 29, 2022
PyTorch implementation for MINE: Continuous-Depth MPI with Neural Radiance Fields

MINE: Continuous-Depth MPI with Neural Radiance Fields Project Page | Video PyTorch implementation for our ICCV 2021 paper. MINE: Towards Continuous D

Zijian Feng 325 Dec 29, 2022
BARF: Bundle-Adjusting Neural Radiance Fields 🤮 (ICCV 2021 oral)

BARF ?? : Bundle-Adjusting Neural Radiance Fields Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey IEEE International Conference on Comp

Chen-Hsuan Lin 539 Dec 28, 2022
[ICCV21] Self-Calibrating Neural Radiance Fields

Self-Calibrating Neural Radiance Fields, ICCV, 2021 Project Page | Paper | Video Author Information Yoonwoo Jeong [Google Scholar] Seokjun Ahn [Google

null 381 Dec 30, 2022
[ICCV 2021 Oral] NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

NerfingMVS Project Page | Paper | Video | Data NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo Yi Wei, Shaohui

Yi Wei 369 Dec 24, 2022
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

NeRF-pytorch NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are

Yen-Chen Lin 3.2k Jan 8, 2023
pixelNeRF: Neural Radiance Fields from One or Few Images

pixelNeRF: Neural Radiance Fields from One or Few Images Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa UC Berkeley arXiv: http://arxiv.org/abs/2

Alex Yu 1k Jan 4, 2023
D-NeRF: Neural Radiance Fields for Dynamic Scenes

D-NeRF: Neural Radiance Fields for Dynamic Scenes [Project] [Paper] D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of

Albert Pumarola 291 Jan 2, 2023
A PyTorch re-implementation of Neural Radiance Fields

nerf-pytorch A PyTorch re-implementation Project | Video | Paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis Ben Mildenhall

Krishna Murthy 709 Jan 9, 2023
[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction Project Page | Paper | Supplementary | Video This reposit

null 331 Dec 28, 2022