Hi, thanks for the efforts and the work is interesting. I have two questions here:

1. Is it the right way to predict segmentations using argmax(integrate(weight * density), slot_dim)? Seems that it could be wrong when occlusion occurs. I am showing a rendered example (the segm on top and the image on bottom) as below. I'm referring to the two cylinders on the right.

2. How can I manipulate clever scenes?

FileNotFoundError: [Errno 2] No such file or directory: './clevr_567_test/00000_sc0000_az00_moved.png'

I'm having this question because a background slot view contains object shadows as shown below. So how will these shadows be during manipulations?

Thanks for your great work. The download link you provide in README needs a Stanford SharePoint account to log in, which makes us cannot access outside. I try to use a personal account or my institution account to access it, but it didn't work.

Could you please solve this or upload it to google drive? Thanks.

Hi, thanks for your great work. I'm trying to reproduce your scene editing baseline. I'm wondering how to get the object moving results in the paper? It's performed by editing slot features before sending them to the decoder or directly manipulating the NeRF volume before the composition.

Hello, I am trying to apply uORF to my own dataset, which currently consists only of multiview RGB images. I noticed in the dataloading code there are many additional files to load aside from the images. It is not clear from the paper which inputs are necessary and what are not.

Could you elaborate on all the different files being loaded in the dataloading code? Like if they are necessary for training the model, if they are for evaluation or debugging, and if they are very useful or not for debugging. This will help me greatly with figuring out if I should generate the corresponding metadata with my custom dataset, or just not use it.

https://github.com/KovenYu/uORF/blob/main/data/multiscenes_dataset.py

A kind request to please clarify regarding nss_scale ? Like I want to generate my own data and train the model on it. How should I determine the nss_scale ?

While using the ./train_clevr_567.sh to train the model on the clever 567 scene, the perceptual loss is zero. Is that the usual behaviour or is it due to something elsle ? Moreover when i modify the shell script to be the trained on uorf_gan model and train_with_gan.py then this is the following output

Reconstruction loss is non zero, otherwise all other losses are zero for the first epoch. Is this fine ?

https://github.com/KovenYu/uORF/blob/d5047198419a64932d47a7df29ea8662979e8b4b/models/model.py#L48

here like from paper i get that it is for getting information in 4 directions ? but what are the 4 directions being talked about ? is this like front and back of the camera? but here as we are rendering foreground objects in viewers view point we only need information in the front of the camera?

Hi, Koven, I wonder if there are the ground truth segmentation masks provided in the dataset. By the way, I want to know how can generate the segmentation labels when generating my own dataset.

Setting the --gpu_ids -1 fails at the moment due to some .cuda() statements. Replacing these with .to(self.device) allows training/ evaluation on a CPU. This can be useful for evaluation on machines without a GPU, or for downloading model weights on cluster login nodes without GPU.

When running bash scripts/eval_nvs_seg_clevr.sh, I got the following device mismatch error:

Traceback (most recent call last):
  File "test.py", line 32, in <module>
    model.test()           # run inference: forward + compute_visuals
  File "/home/.../uORF/models/base_model.py", line 104, in test
    self.forward()
  File "/home/.../uORF/models/uorf_eval_model.py", line 135, in forward
    frus_nss_coor, z_vals, ray_dir = self.projection.construct_sampling_coor(cam2world, partitioned=True)
  File "/home/.../uORF/models/projection.py", line 63, in construct_sampling_coor
    pixel_coor = self.construct_frus_coor()
  File "/home/.../uORF/models/projection.py", line 43, in construct_frus_coor
    z_cam = depth_range[z_frus].to(self.device)
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

This PR fixes this by ensuring that depth_range is moved to the correct device.

Dear authors,

I saw your config supports adding multiple gpus with "--gpu_ids", however the implementation seems doesn't support it. Have you tested training with multiple gpus as well? Thanks

Thanks for your nice work. Is it possible to fit this method into real scene captured by my phone? E.g., I have captured some photos, and then run COLMAP to get camera pose and other parameters. Finally, I put these in uORF. Can I get acceptable object radiance fields? Did you have a try? Or do you have any tips?