I appreciate your sharing of the great work. I have a question. I have been trying to reproduce your work. On the LLFF dataset, however, I can't obtain a reliable warping image by using the depth maps you provided on this website. The warping error seems critical when comparing the warped and target images. Although I aligned with your warping code, it doesn't work. I also test the image warping on the Blender dataset and it works well. Is there an additional process such as depth scaling? If so, please let me know.

Hi, thank for your exciting work,but When I tried to train in the room scene, I had the following problems when loading the VIT model, could you give me some suggestions?

~/NeRFs/SinNeRF$ python train.py --dataset_name llff_ray_patch_1image_proj --root_dir data/nerf_llff_data/room --N_importance 64 --img_wh 504 378 --num_epochs 3000 --batch_size 1 --optimizer adam --lr 2e-4 --lr_scheduler steplr --decay_step 1000 2000 --decay_gamma 0.5 --exp_name llff_room_s4 --with_ref --patch_size_x 63 --patch_size_y 84 --sW 4 --sH 4 --proj_weight 1 --depth_smooth_weight 0 --dis_weight 0 --num_gpus 4 --load_depth --depth_type nerf --model sinnerf --depth_weight 8 --vit_weight 10 Namespace(N_importance=64, N_samples=64, angle=30, batch_size=1, chunk=32768, ckpt_path=None, dataset_name='llff_ray_patch_1image_proj', decay_gamma=0.5, decay_step=[1000, 2000], depth_anneal=False, depth_smooth_weight=0.0, depth_type='nerf', depth_weight=8.0, dis_weight=0.0, dloss='hinge', exp_name='llff_room_s4', img_wh=[504, 378], load_depth=True, loss_type='mse', lr=0.0002, lr_scheduler='steplr', model='sinnerf', momentum=0.9, nH=32, nW=32, nerf_only=False, noise_std=1.0, num_epochs=3000, num_gpus=4, optimizer='adam', patch_loss='mse', patch_size=-1, patch_size_x=63, patch_size_y=84, perturb=1.0, poly_exp=0.9, prefixes_to_ignore=['loss'], proj_weight=1.0, pt_model=None, repeat=1, root_dir='data/nerf_llff_data/room', sH=4, sW=4, scan=4, spheric_poses=False, use_disp=False, vit_weight=10.0, warmup_epochs=0, warmup_multiplier=1.0, weight_decay=0, with_ref=True) Using cache found in /home/zhangzhongwei18/.cache/torch/hub/facebookresearch_dino_main Traceback (most recent call last): File "train.py", line 19, in system = SinNeRF(hparams) File "/home/zhangzhongwei18/NeRFs/SinNeRF/models/sinnerf.py", line 148, in init self.ext = VitExtractor( File "/home/zhangzhongwei18/NeRFs/SinNeRF/models/extractor.py", line 22, in init self.model = torch.hub.load( File "/home/zhangzhongwei18/.custom/cuda-10.2-cudnn8-devel-ubuntu18.04-pytorch1.8.0_full_tensorboard/envs/sinnerf/lib/python3.8/site-packages/torch/hub.py", line 404, in load model = _load_local(repo_or_dir, model, *args, **kwargs) File "/home/zhangzhongwei18/.custom/cuda-10.2-cudnn8-devel-ubuntu18.04-pytorch1.8.0_full_tensorboard/envs/sinnerf/lib/python3.8/site-packages/torch/hub.py", line 430, in _load_local hub_module = _import_module(MODULE_HUBCONF, hubconf_path) File "/home/zhangzhongwei18/.custom/cuda-10.2-cudnn8-devel-ubuntu18.04-pytorch1.8.0_full_tensorboard/envs/sinnerf/lib/python3.8/site-packages/torch/hub.py", line 76, in import_module spec.loader.exec_module(module) File "", line 783, in exec_module File "", line 219, in call_with_frames_removed File "/home/zhangzhongwei18/.cache/torch/hub/facebookresearch_dino_main/hubconf.py", line 17, in import vision_transformer as vits File "/home/zhangzhongwei18/.cache/torch/hub/facebookresearch_dino_main/vision_transformer.py", line 24, in from utils import trunc_normal ImportError: cannot import name 'trunc_normal' from 'utils' (/home/zhangzhongwei18/NeRFs/SinNeRF/utils/init.py)

Hi, there！Sorry to reply you so late, I have been reading your code and running experiments recently.

But I ran into some small problems as follows,

1. When running the test code, the following code will make an error

python eval.py  \
       --dataset_name blender_ray_patch_1image_rot3d  \
       --root_dir ./synthetic_SinNeRF/lego/  \
       --N_importance 64 --img_wh 400 400 --model nerf \
       --ckpt_path ./ckpts/lego_s6_4ft/last.ckpt \
       --timestamp test

When the default value of the --split is test, an error will be reported here. note that frame https://github.com/VITA-Group/SinNeRF/blob/6f101f924fe9ba7793df5a9bbc52b2c82423e251/datasets/blender_ray_patch_1image_rot3d.py#L540

2. The dtu file you uploaded is missing a part or the code is wrong? https://github.com/VITA-Group/SinNeRF/blob/6f101f924fe9ba7793df5a9bbc52b2c82423e251/datasets/dtu_proj.py#L433-L434

3. loss nan problem I am running the latest code on an RTX 3090, 24G, and environments are created using environment.yaml, but is still OOM, so I adjusted the patch_size, precision and --sH, --sW according to the README.

I set precision=16, and --sH, --sW remain the same(=6).

I found that the loss=nan problem will appear when the patch_size is too small, such as patch_size=8 or patch_size=16, even patch_size=32.

It works(without loss=nan) when patch_size=50, but that's not a good number is it?

I would be grateful if you could provide advice on how to deal with this. Thank you !

Hi! Thanks for your great work and implementation!

I'm currently trying your code on nerf_synthetic (lego) and dtu (scan4), while the results after the two-stage training are confusing.

Specifically, the evaluation psnr for lego is 20.6 by the end of step 1, while it drops to 14.9 after step 2 training. The same thing happens to dtu_scan_4, where the evaluation psnr is around 15.0 after step 1 and drops to 11.8 after step 2. The visualization results for lego are as below.

I'd appreciate it if you could provide some idea on this phenomenon and how to fix this problem. Thank you!

lego after step 1 lego after step 2

I'm sorry to bother you. I'm having some problems running your code on my own machine and I hope you can help me out (I'm using 4 A5000 cards for training. Other than that, all other parameters are default). 1, in the NeRF synthetic dataset on step1 training, in the training to more than 1500 steps, all the losses become 'nan', is this normal? 2, after this situation of 1, does it mean that the model has been trained? Should I stop the training? 3, There are several different pth files in the 'ckpts/lego_s6' folder, which one should I choose as the training weights for the second step? 4、You mentioned supplementary material in your paper, but I didn't find the relevant link, can you provide it? I am looking forward to your reply, thank you very much.

Dear author Thank you for your great work. And I am trying to incorporate your work into my pipeline. I found a problem that you use forward warping to get the new RGB information and the new depth. But according to my understanding, forward warping will create holes and loss accuracy. Please point it out if I am wrong. If you stick to the plan I mentioned, how do you solve the problem related with forward warping? Thank you so much!

Thanks again for your great work. Did you provide testing poses for the result videos somewhere in your code? If so, please let me know where I can find it. Thanks for your answer in advance.

Hi, thanks for the great work!

I want to know how you got the depth maps in your datasets. As far as I know, some original datasets like LLFF do not provide depth information.

Thanks!

Hi, Thanks for releasing the code of the awesome work. I am curious about the decay strategy of stride in Progressive Strided Ray Sampling. However, I can not find the correct place in the code. It seems that the stride is controlled by sW,sH in Dataset class but sW,sH are not modified during training. The same problem happens for the vit_weight and dis_weight, which are weights of global structure prior loss and local texture guidance loss.

The fragment in the paper about thedecay strategy of stride:

The fragment in the paper about the annealling of loss weights:

Hello! I have already read your paper and code. But I have something curious about the overall loss function. Because I don't know enough so sorry

you mentioned on paper : we reduce the weight of global structure prior λ3 and increase the weight of local texture guidance λ2. => I don't know it is definitely right, Since you train twice, does that mean that each lambda shrinks and grows between experiments? ..... In all our experiments, λ1, λ2, and λ3 are initialized to be 8, 0.1, and 0, respectively. => But code is not matched to parameter setting in your code. ..... During the training process, we gradually decrease λ2 to 0 and increase λ3 to 0.1 with a linear function. => I don't know what it means, so can you tell me about this and how can I find lamda weight on your code?

Hi, Thanks for your amazing work. I wonder how to generate your depth map of LLFF dataset? Since for LLFF dataset, you provide depth of 5 scenes, could you please provide other 3 scenes? Or method for produce same depth?

Hello! I'm currently doing some research on NeRF, and I found your work SinNeRF. It's an awesome work, and I download the code (in this repository). While reading the code, I found that you use kornia.losses.inverse_depth_smoothness_loss to calculate the "self-supervised inverse depth smoothness loss" which you describe in the paper (equation 4) . The conflict is that, while kornia.losses.inverse_depth_smoothness_loss using the first-order gradient of the RGB Image according to the official document and its source code, the equation in paper use the second-order gradient of RGB Image. Thanks for any advice and help.

Thanks for sharing great work. You have provided one pretrained model, 'room.ckpt'. Do you have any plan to share other models as well? If possible please share other ckpts trained on synthetic 360 and DTU. Thank you so much in advance.