I've trained my own nerf model using 80~100 real images successfully, it takes 8 hours for 50000 iters (probably my GPU not so good), so i think it will take four times hours for 20k iters, it's too long.

And i have no idea about how to use the trained model, because it's not like most general deep learning models. In my opinion, when we train a model for a specific object, then this model can only be tested for images which similar to the train set. so what can we do with our own trained model? all i know is that we can use it to extract mesh model.

My goal is to build a 3D model with some 2D images, i don't konw whether this repo can achieve it or not. The biggest problem is how to use depth map to add color to the extract mesh model. Can you show some guideline for me? I'll be very appreciate.

From what I can decipher, the pose_bounds.npy contains 3x5 pose matrices and 2 depth bounds for each image. Each pose has [R T] as the left 3x4 matrix and [H W F] as the right 3x1 matrix.

However I get confused by the functions poses_avg and recenter_poses. What do these functions do and why?

I checked the original code but there aren't these averaging or recentering.

When in NDC space, the predicted depth has range 0~1. How to translate that into real depth? We know that predicted depth=0 means that it's at the near plane, which is at real distance 1.0, how about predicted depth=1? In the formula it corresponds to infinity, but in reality, do we translate that into the real farthest depth provided by COLMAP? And how about the values that are between 0 and 1?

I ask this question because I want to reconstruct the LLFF data in 3D. Using the predicted depth in 0~1 gives something visually plausible but I think it is mathematically wrong, we need to convert it to real depth.

I tried to train on my own 360 inward-facing scenes, however, there is a huge portion of noise in the output: It doesn't disappear no matter how many steps I train. I follow the suggestion in the readme and add --no_ndc --spherify --lindisp, the other settings are same as fern. I suspect that it is due to the fact that input images has many arbitrary backgrounds, which deteriorates the training? For example some of the training images: These arbitrary backgrounds are inevitable unless I have a infinite ground or an infinitely large table... What's your opinion on this problem? Is it really due to the background or any other reason?

I read the depth map in synthetic test file by d[u][v] = (255-depth[u][v])/255, but the final results seems not consistent in multi-view. So how can I get the accurate depth map?

dirs = np.stack([(i-W*.5)/focal, -(j-H*.5)/focal, -np.ones_like(i)], -1)

'i','j' is the pixel coordinates, why this is the ray direction? Are there some assumptions?

Thanks a lot!

hello, i'm a novice in this field. I was confused about two functions below, they seem to perform transformation between coordinates, could you give more details?

def render_path_spiral(c2w, up, rads, focal, zdelta, zrate, rots, N):
    render_poses = []
    rads = np.array(list(rads) + [1.])
    hwf = c2w[:,4:5]
    for theta in np.linspace(0., 2. * np.pi * rots, N+1)[:-1]:
        c = np.dot(c2w[:3,:4], np.array([np.cos(theta), -np.sin(theta), -np.sin(theta*zrate), 1.]) * rads) 
        z = normalize(c - np.dot(c2w[:3,:4], np.array([0,0,-focal, 1.])))
        render_poses.append(np.concatenate([viewmatrix(z, up, c), hwf], 1))
    return render_poses

def viewmatrix(z, up, pos):
    vec2 = normalize(z)
    vec1_avg = up
    vec0 = normalize(np.cross(vec1_avg, vec2)) #np.cross叉积
    vec1 = normalize(np.cross(vec2, vec0))
    m = np.stack([vec0, vec1, vec2, pos], 1)
    return m

hope you can help me. Thanks!

I followed the instructions to obtain the lego scene from blendswap and I parsed the json files to get the poses and FOV. The poses look correct, but two pieces of info seem to be missing. One is that the scene doesn't appear to be the correct scale. The cameras are within the object without scaling by a factor of 25 or so. The second is that the lego bulldozer scene has a controllable bucket and the renderings in NeRF clearly use a setting other than the default in the blendswap scene.

Do the authors still have the original blender files they used to render these scenes? The licenses seem fairly permissive. It would be great to use these as a starting point for new experiments, but be able to modify them.

Hi, Thanks for your great work of nerf ! Actually, Using colmap to estimate camera extrinsic spends much time (the installation and runing reconstrusction code...). I tried to put a chessboard in my own llff data and then calibrate it by using Opencv. However, the camera parameters could not be correctly used in nerf. I think may be Opencv and colmap have different coordinate systems which makes the camera extrinsic in Opencv can not be used in the code. If you know the answer about how to transform the extrinsic, please tell me. THANK U very much !!

Hi, I just want to confirm that the depth images in the testing dataset of nerf_synthetic are the ground truth or not?

If they are ground truth, could you please also release the depth images for the training and validation datasets?

Many Thanks!

Hi,

I was trying to rewrite the NeRF from scratch to understand the technique better and during the implementation I encountered a concern which I need a help. I am new to computer graphics area so there may be a misunderstanding from my end. Please let me know if that's the case.

If I understood the paper correctly, In NeRF we are supposed to calculate the pixel colors of an image by integrating over the ray sample point colors (c) and volume density values (σ) given by the MLP using the provided formula in the paper.

Since we cant integrate, we use below formula to calculate the estimated ray color.

In the provided implementation of this repo, we first calculate the difference between sampled points (δ). Then we calculate the a value called alpha = 1.0 - tf.exp(-act_fn(raw) * dists) which I think is equivalent to 1 − exp(−σiδi) part of above formula.

Then comes my problem.

weights = alpha * tf.math.cumprod(1.-alpha + 1e-10, axis=-1, exclusive=True) rgb_map = tf.reduce_sum(weights[..., None] * rgb, axis=-2) # [N_rays, 3]

As per my understanding weights variable is equal to the Ti(1 − exp(−σiδi)) part of the formula (Not Exactly). But shouldn't it be implemented roughly like below according to the paper?

alpha = 1 - np.exp(-volume_density * delta)
T_i = np.exp(-np.cumsum(volume_density * delta))
rgb_map = np.sum(T_i*alpha*colors, axis=1)

why this difference exist? Is my understanding is wrong or am I missing something?

Hi nerf team. I have a question.

I tried the demo below https://colab.research.google.com/github/bmild/nerf/blob/master/tiny_nerf.ipynb

And,successfully created a demo video. Next, I want to generate a video from my own images, is that possible? please tell me if there is a way.

Colab removed TensorFlow 1 support, so I added install commands to make it work. I also added a command to install a needed video module.

If the diff is too messy, you can simply copy and paste this code:

# %tensorflow_version 1.x
#
# TensorFlow 1 is deprecated in Colab
# https://stackoverflow.com/questions/73215696/did-colab-suspend-tensorflow-1-x

!echo y | pip uninstall tensorflow
!pip install tensorflow-gpu==1.15
!apt install --allow-change-held-packages libcudnn7=7.4.1.5-1+cuda10.0

# required for video below
!pip install imageio-ffmpeg

Hi! Thanks for your work greatfully：）

I’m now trying to use Nerf on my 360° captured real scene. Using flag --no_ndc --spherify --lindisp. Similar config file as in fern.txt you provided, but got terrible result...

To get a reasonable result, should I use your config for synthetic data like lego.txt?

Hi, I tried to change the L1 supervision to SSIM + L1 supervsion (0.85 * L_ssim + 0.15 L_l1), but failed to get good results. My motivation is that the original L1 supervision only focuses on a single pixel while discarding the local relationship of pixels, thus maybe better performance can be achieved by adding a structural loss like SSIM into supervision. I calculate the SSIM loss by sampling a batch of patches instead of pixels during training (since the memory consumption increases fast when the patch length is large, I only tried path length = 3, e.g. 9 pixels). I am wondering why the performance gets worse by adding the SSIM loss. Any discussion is welcome.

Thanks!

From your supplementary material, I see that "Note that, as desired, t' = 0 when t = 0. Additionally, we see that t' → 1 as t → ∞." And I think it considers depth instead of disparity.
Is not t' sampling from 0 to 1 linear? Why do we need do one more operation-inverse the depth?

What does "using z dimension representing inverse depth" mean?