I am using NDC space and I have depth values collected from a sensor, should I normalize these values between 0 and 1 or should I convert them into NDC space? How should we calculate the depth loss while using NDC space as the depth_col value comes between 0 and 1 in NDC space but the target_depth is bigger than 1.

Thanks.

Hi Authors, I had a question about the implementation of the depth loss. The depth calculated in the colmap depth loader seems to be the z value of the point in camera frame, but the depth computed from the NeRF seems to be the distance of the point along the ray's direction.

What am I reading incorrectly here?

Hi~ Thank you very much for the open source code and paper

I tried to train a model with default config，but I got poor performance on forn_2v.I strictly followed the instructions to start the reproduction experiment，However, my experimental results are quite different from the released results； result of mine： log of last iter：

Am i missing something ? Really appreciate for your help.

Hi, thanks for your great work! I have the question about why to recalculate weights when calculating sigma_loss. While we already have the weights in the forward here.

Hi, thank you for your great work.

I use

python run_nerf.py --config configs/fern_2v.txt --render_only" 

command with the pretrained model, it gave me these results

https://user-images.githubusercontent.com/45889437/202093543-8c02a2f1-8989-48d7-8ff5-ead60cdbdf36.mp4

https://user-images.githubusercontent.com/45889437/202093571-40752187-6e50-4847-912f-cbdd1ac6dbe9.mp4

while when I re-trained the model from scratch with this command

python run_nerf.py --config configs/fern_2v.txt --weighted_loss 

and render it with my trained model, it gave this following results

https://user-images.githubusercontent.com/45889437/202093604-a700141a-77a3-4e85-a25d-b842d8cf902e.mp4

https://user-images.githubusercontent.com/45889437/202094729-efc05d95-0c5c-4d06-8b8a-1505e3b295b4.mp4

Do you have any clue ? Is the sigma-loss necessary in order to reproduce the same quality with the given-pretrained model? Thanks

Hi,

Thank you for the great work.

If my understanding is correct, the depth D of a 3D point is its Z value in a camera's local coordinate system. It is different from the ray distance t, which is the distance from the point to the camera center. In fact, D = sin(θ) × t, where θ is the angle between the camera plane and the ray. Therefore, even in an ideal case, the distribution of h(t) and D are not expected to be the same.

However, in section 3.2, I see you minimizing the KL divergence between these two distributions. Wouldn't that cause some problems? Or did I miss something?

I am new to nerf, please forgive me if the answer is trivial.

Thank you

Hi, thanks for your nice work!

I found an issue while trying to use your KL-divergence loss. In loss.py, almost half of alpha values become zero due to negative raw density values. alpha = raw2alpha(raw[...,3] + noise, dists) # [N_rays, N_samples]

This produces zero weights for such points, and ends up passing a bunch of zeros into the log function. weights = alpha * torch.cumprod(torch.cat([torch.ones((alpha.shape[0], 1)).to(device), 1.-alpha + 1e-10], -1), -1)[:, :-1] loss = -torch.log(weights) * torch.exp(-(z_vals - depths[:,None]) ** 2 / (2 * err)) * dists

How did you deal with this problem in your experiments? (eg. perhaps by masking out those points while computing the loss, or by using a softplus activation in raw2alpha instead of relu?)

Thanks in advance!

Hi, thank you for the work.

I have a question about the sparse 3d points given known cameras. I have read this issue and this issue.

I also carefully read this link of colmap. There are multiple steps, like image.txt, cameras.txt, and point3D.txt. I would like to know how to prepare these to align your work to better reproduction for comparison.

I think this is the way (with known camera poses) you prepare NeRF real (LLFF) and DTU since we need unified poses for training and test images. The steps are as follows:

(1) estimate poses with training and test images (because we need a unified coordinate for training and test). (2) With camera poses of training images, we generate sparse 3d points. Note, we cannot access test images to generate sparse 3d points.

In the code, the paper only provides code for python imgs2poses.py <your_scenedir>

It would be better if a script like python imgs2poses.py <your_scenedir> <your camera poses> <your camera intrinsics> is provided. I would like to know how to prepare these to align your work to better reproduction for comparison.

Thank you.

Hello, I try to run your code with my own data, but I found the depth loss is pretty large (0.25). I'm not sure if it is because the generated depth is not right in my own data, so I want to know the depth loss's value in your training.

Hi, thanks for your neat and excellent work! However, I am confused about the implementation of the weight for depth and RGB rendering. In the code implementation, it seems that the weight is calculated by the product of T(t) and alpha(t) (https://github.com/dunbar12138/DSNeRF/blob/168a13edd8a08396e3ad775283eeb65145317205/run_nerf_helpers.py#L374) while it is formulated as the product of T(t) and sigma(t) in the paper.

Hi,

I noticed that you compute depth in input views in https://github.com/dunbar12138/DSNeRF/blob/main/load_llff.py#L371 but the depth given by NeRF model is in cannonical view https://github.com/dunbar12138/DSNeRF/blob/main/run_nerf_helpers.py#L377

Shouldn't both the depths be brought to same view before computing depth loss? Please let me know if I missed something

Hi~ I'm going to apply it to the new data instead of the dataset that provided your great work, dsnerf. So I'd appreciate it if you could tell me how to generate the 'train_depths.npy' file and how to make the information (depth, coord, error).

When I run my images through COLMAP, I get a Sparse directory and can run imgs2poses which generates poses_bounds.npy.

How do I go about converting my test and training images to .npy format and how do I generate bds.npy, test_poses.py, train_poses.py, etc...

Thanks for any assistance.

Hello, I recently tried to add depth KL scatter loss to my own project, and found the core code corresponding to this section was then found to be here https://github.com/dunbar12138/DSNeRF/blob/d9deb70ac1464348aebbe5e32fc98aeca1920f70/loss.py#L87 But in my actual use I found a problem that the weights will often be 0 So when calculating the loss, if you put the weights directly into the log function, this can easily result in results with inf But I looked at the code and found that there was nothing to prevent this situation

So I would like to ask guys if anyone has encountered this situation, and I think a very small number can be added to the weights to prevent this as follows:

-torch.log(weights+1e-10) * torch.exp(-(z_vals - depths[:,None]) ** 2 / (2 * err)) * dists

Sorry for disturbing you and hope to receive an answer, thans a lot (^▽^) !

Greetings! I find the config files that you release now are actually neglecting many important training options, for example, whether use weighted_loss or not (refer to this issue #30). Besides, the pretrained logs you provided in download_models.sh are actually missing the args.txt and config.txt. Since these hyper-parameters are vital to reproduce the reported performances in paper, could you please completely upload these files for each scene and each dataset? Or it is really hard for for future works to make experimental comparisons with DSNeRF. Thanks!

Hi, Thank you for sharing your great code. I'm glad to learn about DSNeRF with your kind and detailed explanation

I have a question about sigmaLoss with paper.

I guess sigma loss is the ray distribution loss in the paper notation by h(t). From the paper, h(t) = T(t)*σ(t) is defined. But the code in class SigmaLoss from loss.py, I see

        # sigma = F.relu(raw[...,3] + noise)
        alpha = raw2alpha(raw[...,3] + noise, dists)  # [N_rays, N_samples]
        weights = alpha * torch.cumprod(torch.cat([torch.ones((alpha.shape[0], 1)).to(device), 1.-alpha + 1e-10], -1), -1)[:, :-1]
        
        loss = -torch.log(weights) * torch.exp(-(z_vals - depths[:,None]) ** 2 / (2 * err)) * dists
        loss = torch.sum(loss, dim=1)
        
        return loss

loss = -torch.log(weights) * torch.exp(-(z_vals - depths[:,None]) ** 2 / (2 * err)) * dists Here, the loss function is defined by weights and $T_i$. I think that from the paper's h(t) definition, the loss should be defined by loss = torch.log(h) * torch.exp(-(z_vals - depths[:,None]) ** 2 / (2 * err)) * dists where h = sigma * torch.cumprod(torch.cat([torch.ones((alpha.shape[0], 1)).to(device), 1.-alpha + 1e-10], -1), -1)[:, :-1] But, the actual code is not. Can I guess why this code is different from the DSNeRF paper? Additionally, can I ask why the '-' term is added in front of torch.log(weights)?

Thanks

Hello,

Thank you very much for your great implementation! I have successfully trained the model on my own data, but am now running into a GPU memory problem when trying to render a novel view. Is there any way to decrease this memory load while "rendering only"? It would generally be great to get a bit more information on the different variables in the config file.

Thank you very much!