Hi, thank you for sharing the cool work. I am wondering how to derive the matrix warping formula in section 3.1 i.e

Because according to my own derivation, the result is

Are there any derivation premises that I have not considered? Thanks in advance

Hi, I really appreciate your work. An impressive idea. I want to ask a question regarding the 3D CNN architecture:

1. It's written, that CBR3D0 takes T which is cost volume of shape (BATCH_SIZE, CHANNELS (32), N_DEPTH, HEIGHT / 4, WIDTH / 4) and I which are concatenated images, as I guess (BATCH_SIZE, 9, 1, HEIGHT, WIDTH). So, there appears to be a shape inconsistency. Could you, please, clarify this one, if possible.
2. It also states, that CBT3D2 takes outputs from CTB3D1 and CBR3D2, which is a total of 48 channels instead of 64. Is it a typo? Thanks in advance for your reply!

Hi, I used your code in render.ipynb and tried to evaluate the pretrained model on LLFF, but I found the metrics are significant lower than that reported in the paper, for example: for the room scene, the result PSNR are only 22.99, but in the paper it is 26.95.

As you said in #7 ,

we think building volume in the target view may provide a better depth quality but is not an efficient way to do the free-viewpoint rendering

I also feel that the fact that the cost volume must be in some reference view has many limitations. Leave alone view extrapolations, how does your method perform when you move the camera closer to the scene (not zoom in but physically place the camera closer to the scene)?

Currently, the LLFF scenes are all captured roughly at the same distance to the scene, so interpolating at this distance seems great as your video in the readme. However I wonder how it performs in the situation I describe above. With the current cost volume, I can think of two problems:

1. Although this novel view is still view interpolation (it lies inside the frustums of all reference views), the view to the nearest reference view is very far.
2. The novel view lies inside the cost volume, a situation that is not seen at all at training time.

To my knowledge, traditional MPI methods cannot handle this kind of situation well either. NeRF performs excellently without problem since it reconstructs the whole scene in 3D without relying on reference views in test time. Following is an example where I move the camera very close to the scene (left is the nearest reference view and right is NeRF synthesized result).

Pardon me for not being able to run your code myself. I didn't find an easy way to run your code with specifying a pose...

Hello, thanks for your excellent work! In the original paper, it seems that you have claimed that during finetuning, MVSNerF is free from the 2d feature extractor and the 3d cnn (Correpsonding to the MVSNet class of the code), which means the MVSNet doesn't need to be optimized during fintuning.

In introduction

In essence, the encoding volume is a localized neural representation of the radiance field; once estimated, this volume can be used directly (dropping the 3D CNN) for final rendering by differentiable ray marching.

In section 3.4

Note that, we optimize only the encoding volume and the MLP, instead of our entire network

However, intrain_mvs_nerf_finetuning_pl.py and models.py. It seems that the MVSNet also needs to be optimized during finetuning, which confuses me.

Could you please explain the difference of the paper and the code? Many thanks!

In dtu.py:

if self.split=='train':
    ids = torch.randperm(5)[:3]
    view_ids = [src_views[i] for i in ids] + [target_view]

It seems that only a small number of images are used for training in the dtu dataset?

Hi. src_grid = src_grid.view(B, D, W_pad, H_pad, 2) in function homo_warp. The shape of src_grid seems to be (B, DHW, 2), so is there something wrong with the above code?

Hi, it's very kind of you for releasing the code of the paper, and there are some questions about code details and abnormal phenomenon:

As is shown in the first image, I am bit confused of the codes in 140-144 lines of ./data/dtu.py. The reference should be the first element of "view_ids", the in codes in 140-144 lines you have put in the last position. Is something wrong with my understanding?

And again in the codes in 233-234 lines of ./train_mvs_nerf_pl.py, here appears two notions, tgt and ref. So may I ask what exactly src, ref, and tgt stands for?

Is the "with_depth" option means that we have got the depth map of dataset and we should not activate it if we do not get the depth map?

Last qusetion, when I train with my own dataset, nan appears after a few iterations. Have you ever met with the same problem?

Looking forward to your answers. Thanks for your work again.

Hi, thanks for your great work. I would like to make an enquiry into how dtu_pairs.txt file was created (not pairs.th file) and especially into what the floating point values mean in each line.

Thank you

Hi, I have pull down your newest code. However, I cannot run the finetune script for you have not define the img_scale in train_mvs_nerf_finetuning_pl.py

Hi, I have some questions concerning accuracy in the following:

1. How to evaluate the accuracy for the 16 DTU? I have changed the max_len from 10 to -1 in val_dataset. Does the modification work right and Does there exist other methods for evaluating?

2. Except #114 scene in DTU, I cannot implement the accuracy of other scenes (1, 8, 21, 103) as the paper said. Could you please offer the run commands for pretraining and fine-tuning (DTU) to the official accuracy? And I am sure the dataset is right because the #114 also comes from it.

3. By the way, is the newest update(597f5) significant, can the older version (736d9a4b) be restored its accuracy?

Hello, thanks for your great work I wonder when applying mvsnerf on other types of data ( faces, bodys, for example ), what should i do with the extrinsic and the intrinsic in my own data. To be more detailed 1: Do i need to convert the extrinsic matrix to opencv format? If so, do a simple matrix-multiplication between the extrinsic(c2w for my own data) and the blender2opencv matrix is enough ? 2: How should i deal with the scale factor and the downsample factor like what is in ./data/dtu.py. Do i need to perform intrinsic[:2] = intirnsic[:2]*4 when collecting intrinsic, and intrinsic[:2] = intrinsic[:2]/4 when calculating proj mats like in dtu.py, or directly perform dividing 4 without intrinsic[:2] = intirnsic[:2]*4.

Looking for your reply !

Dear author, If I want to use less than 3 images(included) to reconstruct complicated mesh, such as human full body with clothes, which method could do a better job now? pixelNerf, mvsNerf or any other feasible methods?

By the way, will this project develop mesh generation functionality in the future?

Look forward to your reply, thank you.

Hi there,

Thank you for the great work! When I try to follow your code, I have a question on the raw2alpha function at https://github.com/apchenstu/mvsnerf/blob/main/renderer.py#L18, which describes the process of transforming volume density to the weights of rendering. However, in original NeRF paper, the distance between ray points should be another factor in the exponential function. In both your paper and your code, it seems that this term is gone.

I tried your original code and I found ignoring the distance does not affect the performance that much. But is there any particular reason that you ignore this term?

Thank you for your time and look forward to your reply!

Hi! Thanks for open sourcing the code of your wonderful work!

I have some questions about the position input x of the NeRF MLP. If I get it right, x is in NDC space. While the same point in the world space may have different 'x' in the NDC space, as the output view changes. So could you share some insights on the effect of x here? Thanks a lot!