Hi, thanks for your code and efforts! I have two questions which about pre-render the Dataset.

1. From the NeRF Dataset we can take photos and get poses through the library directly. How do you pre-render the Replica Dataset which you provided us?
2. How can we pre-render our own dataset for using this semantic-nerf? What should I prepare my own Dataset Many thanks for your reply!

Thank you very much for your work and sharing However, when I download the pre-rendered Replica data, it shows failure at the end of the download. Is there any other way to download it?

Hello! Thank you a lot for your great work! I am planning to use your method for robotic application and I have a question. Do you have any guide how to prepare my own data for training and rendering? As I understood the input includes 2D color images, depth images, camera poses and sematic class? Am I correct? These data should be stored as a dataset?

Thank you in advance!

appreciating your excellent work !!! I want to follow your work, but when I used your most recent code in the replica room_0, I discovered that the void class could not display in the image,but the generated sem.mp4 can display void class normally. Why is this? I recall using your code a few months ago, and the output image showed a void class(color is black). Please give me some advice on how to switch back to the old code. Thanks a lot!!!

Hi,

Thank you for the wonderful work. You said camera poses of Replica are randomly generated in #5 . However, the images rendered seem to be in a smooth trajectory instead of produced from random camera poses. May I ask what is the algorithm used to generate the trajectory?

Thanks!

Hi! Thanks for your great job again! It is really a cool idea to integrate the semantic in the nerf model. And I meet some trouble in the visualization, I can train perfectly now on my computer, but when I try to visualize the mesh there is always error of CUDA out of memory, i try to reduce the chunk and the netchunk to 8 but it still occurs, did you have any idea? FYI I notice that in the visualization you use another test.yaml, is it possible that you share this test.yaml?

Hi, i want to test semantic_nerf model, but is extract colour mesh file for testing. If it is, why neccessary mesh file, is nerf model need to generate mesh model. In conclusion, how can i do testing

The code is wonderful. But I have some trouble generating per scene semantic img from this code. o3d_mesh_canonical_clean.vertex_colors = o3d.utility.Vector3dVector(v_colors/255.0) Could you please give me some advice?

Hello! Thanks for your great work!! I am very interested in your semantic-nerf. And I encountered a problem when I apply other model on your pre-rendered replica dataset. The same parameters performed well on room0, but performed very poorly on scene data like office4. Have you encountered a similar problem? Are the hyperparameters of each scene basically the same?

This is novel view synthesis result of room0:

This is novel view synthesis result of office4:

I want to run on my win10 computer, and the GPU is RTX2060super8G. Maybe GPU is not good enough :) , and I just want to try to run through it.

Traceback (most recent call last): File "train_SSR_main.py", line 225, in train() File "train_SSR_main.py", line 213, in train ssr_trainer.step(global_step) File "D:\Users\admin\Documents\GitHub\semantic_nerf\SSR\training\trainer.py", line 981, in step rgbs, disps, deps, vis_deps, sems, vis_sems, sem_uncers, vis_sem_uncers = self.render_path(self.rays_vis, save_dir=trainsavedir) File "D:\Users\admin\Documents\GitHub\semantic_nerf\SSR\training\trainer.py", line 1156, in render_path output_dict = self.render_rays(rays[i]) File "D:\Users\admin\Documents\GitHub\semantic_nerf\SSR\training\trainer.py", line 703, in render_rays all_ret = batchify_rays(fn, flat_rays, self.chunk) File "D:\Users\admin\Documents\GitHub\semantic_nerf\SSR\training\training_utils.py", line 10, in batchify_rays ret = render_fn(rays_flat[i:i + chunk]) File "D:\Users\admin\Documents\GitHub\semantic_nerf\SSR\training\trainer.py", line 745, in volumetric_rendering raw_coarse = run_network(pts_coarse_sampled, viewdirs, self.ssr_net_coarse, File "D:\Users\admin\Documents\GitHub\semantic_nerf\SSR\models\model_utils.py", line 31, in run_network embedded = torch.cat([embedded, embedded_dirs], -1) RuntimeError: CUDA out of memory. Tried to allocate 1.65 GiB (GPU 0; 8.00 GiB total capacity; 5.00 GiB already allocated; 910.21 MiB free; 5.14 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

After GOOGLE, I know I need to reduce the batch_size or chunk size, but I don't know where PROJECT CODE I need to edit and what reduced value of batch_size and chunk size is proper. And batch_size or chunk size, which one should I choose to reduce first?

thanks !!!!!!

Edit: I'm a beginner in NeRF. Apologize if it's a dummy question:)

Hi,

Thank you for your inspiring work and the code released!

I'm trying running your demo on "room0" of Replica by

python3 train_SSR_main.py --config_file /SSR/configs/SSR_room0_config.yaml

Whenever it comes to validation time, it will not be able to move on and interrupted by "CUDA out of memory". However, I use the same GPU (RTX2080-Ti with 11GB memory) as mentioned in your paper. Also, no matter how much I decrease the numbers "chunk" and "netchunk" in the config file (even decreasing them to 1024*1), this "CUDA out of memory" would happen during validation time.

I'm wondering whether you could give me some advice.

Hello, thanks for your excellent job! Generating a semantic model in a nerf-way is so cool. With the shared codes, I succeeded in training with Replica data and stopped the training when the testing results are perceptually good. However, when generating colored meshes, I met some trouble because no "mesh.ply" file can be found. I wonder where I can get the "mesh.ply" file.