Hi, thanks for your great job! I have the same problem when I run "bash dev_scripts/w_n360/ship_test.sh". It goes with "FileNotFoundError: [Errno 2] No such file or directory" in line 118 of load_blender.py. My datapath is pointnerf/data_src/nerf/nerf_synthetic/ship，and it includes 3 .json files and 3 folders containing some .png pictures. At the same time, the checkpoints folder only include some .pth files. It doesn't seem to contain the saved point cloud.

Could you please tell me where the "point_path" is? Thank you~

Thanks for sharing this work. i am a little confused about the loss item. i know that 'coarse_raycolor' is corresponding to L_render in paper, what is 'ray_masked_loss' and 'ray_miss_loss'? And if i understand correctly, 'zero_one_loss_items' is the L_sparse in paper. but since we already set the neural points as input data and didn't load mvsnet model, can the parameters of mvsnet also be updated? (when i inspect the variable , the only model is "ray_marching") And you set the neural points to nn.parameter to update, any reasons behind it?

Hi everyone, I have a problem when launching bash scripts like "bash dev_scripts/w_n360/chair.sh", the argument 'name' is not detected and the terminal returns me the error: train_ft.py: error: the following arguments are required: --name Did any of you had the same problem?

I run this command to do Per-scene optimize from scratch: bash dev_scripts/w_n360/lego.sh

In the paper, the speed should be 2min / 1K iters. However, on my single 2080Ti, it takes 100min / 1K iters. Anyone meets this problem?

Hello, Thanks for the awesome work. However when I run the Per-scene Optimization on NeRF-Synthetics I found some error below:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Debug Mode ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ /home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/numpy/core/shape_base.py:420: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray. arrays = [asanyarray(arr) for arr in arrays] /home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1646755953518/work/aten/src/ATen/native/TensorShape.cpp:2228.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] dataset total: train 100 dataset [NerfSynthFtDataset] was created ../checkpoints/nerfsynth/lego/*_net_ray_marching.pth ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Continue training from 200000 epoch Iter: 200000 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ opt.act_type!!!!!!!!! LeakyReLU self.points_embeding torch.Size([1, 479862, 32]) querier device cuda:3 3 Traceback (most recent call last): File "train_ft.py", line 1084, in main() File "train_ft.py", line 637, in main model = create_model(opt) File "/mnt/data1/zhaoboming/pointnerf/run/../models/init.py", line 39, in create_model instance.initialize(opt) File "/mnt/data1/zhaoboming/pointnerf/run/../models/base_rendering_model.py", line 369, in initialize self.create_network_models(opt) File "/mnt/data1/zhaoboming/pointnerf/run/../models/mvs_points_volumetric_model.py", line 44, in create_network_models super(MvsPointsVolumetricModel, self).create_network_models(opt) File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points_volumetric_model.py", line 157, in create_network_models params = self.get_additional_network_params(opt) File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points_volumetric_model.py", line 142, in get_additional_network_params self.neural_points = NeuralPoints(opt.point_features_dim, opt.num_point, opt, self.device, checkpoint=checkpoint_path, feature_init_method=opt.feature_init_method, reg_weight=0., feedforward=opt.feedforward) File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/neural_points.py", line 331, in init self.querier = self.lighting_fast_querier(device, self.opt) File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/query_point_indices_worldcoords.py", line 39, in init self.claim_occ, self.map_coor2occ, self.fill_occ2pnts, self.mask_raypos, self.get_shadingloc, self.query_along_ray = self.build_cuda() File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/query_point_indices_worldcoords.py", line 524, in build_cuda """, no_extern_c=True) File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 358, in init include_dirs, File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 298, in compile return compile_plain(source, options, keep, nvcc, cache_dir, target) File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 87, in compile_plain checksum.update(preprocess_source(source, options, nvcc).encode("utf-8")) File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 59, in preprocess_source "nvcc preprocessing of %s failed" % source_path, cmdline, stderr=stderr pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmpeytiys5x.cu failed [command: nvcc --preprocess -arch sm_86 -I/mnt/data1/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/cuda /tmp/tmpeytiys5x.cu --compiler-options -P] [stderr: b'cc1plus: fatal error: cuda_runtime.h: No such file or directory\ncompilation terminated.\n'] end loading

PyCUDA ERROR: The context stack was not empty upon module cleanup.

A context was still active when the context stack was being cleaned up. At this point in our execution, CUDA may already have been deinitialized, so there is no way we can finish cleanly. The program will be aborted now. Use Context.pop() to avoid this problem.

So How can I fix it? Thanks.

Hi , when I redownload the code repo and run from scratch ,It still encounters with the problem of "No such file or directory" when I run scene101.sh ,However it works fine if I run from scratch in the dataset of Nerf Synth. Is there some possiblity that I should change some setting for the scannet scene. https://github.com/Xharlie/pointnerf/issues/7#issuecomment-1060343926

In base_rendering_model, loss = self.l2loss(masked_output, masked_gt) * masked_gt.shape[1], why multiply masked_gt.shape[1]? https://github.com/Xharlie/pointnerf/blob/a614e1d15cc8409e14f80c7b3dd0091939fef758/models/base_rendering_model.py#L560

Hello authors, thank you very much for your paper and for releasing code! I am very excited about this work. I have gotten point cloud data in a ply format somehow, how do I use this data as input to point_nerf. I don't seem to find it in "READEME.md".

Hi,

We evaluated your code on the Tanks&Temples dataset and found that the images outputted by your model have a size (resolution) of 1088x640‬, with a different ratio from the original image size of 1920x1080. It's also shown in your config file that you manually set it as 1088x640‬.

For Table 9 in your paper, you compared the results with NSVF, and the numbers of NSVF were the same as those in NSVF paper, which also claims that "The image resolution is 1920 × 1080" for Tanks&Temples.

Why did you choose the resolution of 1088x640? Were the metrics in Table 9 in your paper calculated with images with a resolution of 1088x640 or 1920x1080?

Thanks.

Hello,

Thanks for the awesome work. However, I am not able to download the DTU training dataset from the gdrive link shared in the repo. It seems that the link is dead. Would it be possible for you to reactivate the link?

Thanks, Aditya Vora

Thank you for your work! There is a value error when I test the per-scene optimized Point-NeRFs. For example, run: bash dev_scripts/w_n360/chair_test.sh

I checked the line 92 in visualizer.py and find the total_step is not a integer but a string. Therefore, the filename cannot be generated and will bring this error because of mismatched variable type. You can fix this bug by converting total_step to int. Thank you for your time.

(point-nerf) zzh@zzh-pc:~/Projects/NeRF/pointnerf$ bash dev_scripts/w_n360/lego_cuda.sh dev_scripts/w_n360/lego_cuda.sh: line 291: 9526 Segmentation fault python train_ft_nonstop.py --experiment $name --scan $scan --data_root $data_root --dataset_name $dataset_name --model $model --which_render_func $which_render_func --which_blend_func $which_blend_func --out_channels $out_channels --num_pos_freqs $num_pos_freqs --num_viewdir_freqs $num_viewdir_freqs --random_sample $random_sample --random_sample_size $random_sample_size --batch_size $batch_size --maximum_step $maximum_step --plr $plr --lr $lr --lr_policy $lr_policy --lr_decay_iters $lr_decay_iters --lr_decay_exp $lr_decay_exp --gpu_ids $gpu_ids --checkpoints_dir $checkpoints_dir --save_iter_freq $save_iter_freq --niter $niter --niter_decay $niter_decay --n_threads $n_threads --pin_data_in_memory $pin_data_in_memory --train_and_test $train_and_test --test_num $test_num --test_freq $test_freq --test_num_step $test_num_step --test_color_loss_items $test_color_loss_items --print_freq $print_freq --bg_color $bg_color --split $split --which_ray_generation $which_ray_generation --near_plane $near_plane --far_plane $far_plane --dir_norm $dir_norm --which_tonemap_func $which_tonemap_func --load_points $load_points --resume_dir $resume_dir --resume_iter $resume_iter --feature_init_method $feature_init_method --agg_axis_weight $agg_axis_weight --agg_distance_kernel $agg_distance_kernel --radius_limit_scale $radius_limit_scale --depth_limit_scale $depth_limit_scale --vscale $vscale --kernel_size $kernel_size --SR $SR --K $K --P $P --NN $NN --agg_feat_xyz_mode $agg_feat_xyz_mode --agg_alpha_xyz_mode $agg_alpha_xyz_mode --agg_color_xyz_mode $agg_color_xyz_mode --save_point_freq $save_point_freq --raydist_mode_unit $raydist_mode_unit --agg_dist_pers $agg_dist_pers --agg_intrp_order $agg_intrp_order --shading_feature_mlp_layer0 $shading_feature_mlp_layer0 --shading_feature_mlp_layer1 $shading_feature_mlp_layer1 --shading_feature_mlp_layer2 $shading_feature_mlp_layer2 --shading_feature_mlp_layer3 $shading_feature_mlp_layer3 --shading_feature_num $shading_feature_num --dist_xyz_freq $dist_xyz_freq --shpnt_jitter $shpnt_jitter --shading_alpha_mlp_layer $shading_alpha_mlp_layer --shading_color_mlp_layer $shading_color_mlp_layer --which_agg_model $which_agg_model --color_loss_weights $color_loss_weights --num_feat_freqs $num_feat_freqs --dist_xyz_deno $dist_xyz_deno --apply_pnt_mask $apply_pnt_mask --point_features_dim $point_features_dim --color_loss_items $color_loss_items --feedforward $feedforward --trgt_id $trgt_id --depth_vid $depth_vid --ref_vid $ref_vid --manual_depth_view $manual_depth_view --pre_d_est $pre_d_est --depth_occ $depth_occ --manual_std_depth $manual_std_depth --visual_items $visual_items --appr_feature_str0 $appr_feature_str0 --init_view_num $init_view_num --feat_grad $feat_grad --conf_grad $conf_grad --dir_grad $dir_grad --color_grad $color_grad --depth_conf_thresh $depth_conf_thresh --bgmodel $bgmodel --vox_res $vox_res --act_type $act_type --geo_cnsst_num $geo_cnsst_num --point_conf_mode $point_conf_mode --point_dir_mode $point_dir_mode --point_color_mode $point_color_mode --normview $normview --prune_thresh $prune_thresh --prune_iter $prune_iter --full_comb $full_comb --sparse_loss_weight $sparse_loss_weight --default_conf $default_conf --prob_freq $prob_freq --prob_num_step $prob_num_step --prob_thresh $prob_thresh --prob_mul $prob_mul --prob_kernel_size $prob_kernel_size --prob_tiers $prob_tiers --alpha_range $alpha_range --ranges $ranges --vid $vid --vsize $vsize --wcoord_query $wcoord_query --max_o $max_o --zero_one_loss_items $zero_one_loss_items --zero_one_loss_weights $zero_one_loss_weights --prune_max_iter $prune_max_iter --far_thresh $far_thresh --bg_filtering $bg_filtering

Could you please give more information about how to sample points in order to make sure the sample points are near neural points? Do we need to calculate the distance between every voxels(defined by CAGQ method) and the ray, then pick the nearest voxel? Thank you!

When I run the test code, there are several extra packages to install and I am stuck at the step of installing inplace_abn. I cannot install it by either using pip or directly run setup.py. It seems that the problem is caused by the high torch version.

With torch==1.5.0, inplace_abn can be compiled without error.

When it comes to 1.8.1, it will throw an error about ['ninja','-v'] Some blog article tells that we can manually change '-v' to '--version' but it will throw new errors

g++: error: ~/inplace_abn/build/temp.linux-x86_64-cpython-37/src/inplace_abn.o: No such file or directory
g++: error: ~/inplace_abn/build/temp.linux-x86_64-cpython-37/src/inplace_abn_cpu.o: No such file or directory
g++: error: ~/inplace_abn/build/temp.linux-x86_64-cpython-37/src/inplace_abn_cuda.o: No such file or directory
g++: error: ~/inplace_abn/build/temp.linux-x86_64-cpython-37/src/utils.o: No such file or directory
error: command '/usr/bin/g++' failed with exit code 1

By using torch==1.5.0, we can get these src files to subtitute these vacancies of files. But later, a new error occurs like: ImportError: ~/anaconda3/envs/pointnerf/lib/python3.7/site-packages/inplace_abn-1.1.1.dev6+gd2728c8-py3.7-linux-x86_64.egg/inplace_abn/_backend.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceISt7complexIdEEEPKNS_6detail12TypeMetaDataEv

Then I have no ideas. Anyone who succeeded in the compilation process can help?

In the part H Neural Point Querying from the paper, the evenly spaced 3D grids are mentioned. And the paper says that 'Since these grids in the perspective coordinate are cubic, in the world coordinate, they have shapes of spherical voxels'. Could you please give more detailed information about the perspective coordinate? And may I ask why the grids can have shapes of spherical voxels in the world coordinate? Thank you!

Hi，Charlie！ 谢谢你的工作，当我在使用Colmap创建的点云（或者是像scannet一样从depth中初始化点云）的时候，在train_ft.py中会调用一个nearest_view 函数，我理解的这个函数的目的是计算每个点属于哪一个view。便于后面使用query_embeding函数去提取初始特征。那么我的问题：在倒数第三行这里对“点云方向和相机中心像素点方向的内积“ 与 “点到campos的距离” 进行了一个加权处理。请问为什么这里距离的权重是1/200，您在这里是怎么考虑的呢，适合应用到什么样的数据集上面？我可不可以任意调整这个数。

Hello, thanks for sharing the code for the amazing work.

I was curious to know the colmap settings you use to reconstruct the point cloud for the nerf dataset. Would it be possible to share the script for doing the reconstruction? I checked the links that you provided in the repo, however, it only has the models and not the script to generate those.

Thanks, Aditya