Hello， I came here from the Instant ngp issues “export of 3D model is not clear ...". Is there any way to export clear 3D models to run on general 3D software in this paper?

Thanks to the authors for your contributions. Here I have some questions about relighting:

1. When I run test_relighting.py using milkbox sample with an outdoor image (download from google), but the relighting results seem a littie bad. After checking the code, I found that the relighting results were highly dependent on the setting of args.test_env_maxthres. I am curious about how to choose the env image and set the args.test_env_maxthres to achieve the better relighting results?

The outdoor image we used while tesing the milkbox sample:

The relighting results:

2. In section 3.5 of the paper it is introduced that the purpose of Rendering Networks is to estimate the lighting of each input image and the material properties of the object, but the self.env_lights in class NeROICRenderer keeps the value unchanged since the initialization. I am curious about how to the network learning the various illumination conditions?

I modifid train.py\ generate_normal.py\dataset.py to cpu device type。 Then I execute code python train.py \ --config configs/milkbox_geometry.yaml \ --datadir ./data/milkbox_dataset. But there not generate 'epoch=29.ckpt' file

Your dataset structure follows common LLFF, and I understand that the main issue with custom datasets and LLFF is the lack of file names being presented to the model. I had similar issues with nvdiffrec and a simple list solves any number of memory leaks that happen when loading images/files rejected by colmap.

a "view_imgs.txt" is pretty important I'd think, and I'm glad some of the example datasets use a poses.npy -I do not understand the reasoning to use remove.bg masks, construct datasets with another .db file, and pickle a list of files (instead of a readable .txt) that users might want to edit when making their own sets. https://github.com/snap-research/NeROIC/blob/e535d50077ab57c2f39ae55543c8543793857676/dataset/llff.py#L243 when I am trying to handle masks, in their own separate folder, would I want them just to be a b/w image out of any number of salient object matting repos, or images specifically from remove.bg of a separate size (to then use bicubic filtering on), only in the alpha, and containing an entire unused image?

There are 0 technological limitations on making a dataset renderable and testable in instant-ngp, meshable in nvdiffrec (and this repo), optimizable in AdaNeRF and R2L, and still be created from a video shot on a phone. If you're planning on making a dataset creation guide, please dont use removebg filenames, dont use new db files, dont use non user-readable lists of files (it takes one extra line to parse a .txt file), and have support for traditional b/w masks. all that's needed is /images, /masks, imgs.txt, and a poses.npy (pts seems like it's to build a bounding box and isn't in all your example sets) lowering that barrier allows anyone who knows how to run a script to make datasets, and it's WHY instant-ngp worked, anyone could try it out with ffmpeg and a script. Forks are being made to test datasets made from my colmap2poses script, if a simple colmap2NeROIC script is needed to read colmap data I can make a push with a more forgiving LLFF dataloader and said script.

I found some hyperparameters in configuration files inconsistent with those reported in the original paper. For example, lambda_tr is 1 in config while 0.01 in the original paper. Which one should I refer to?

The masks in provided data seem perfect. I am working on my own data and wondering if it can work with noisy foreground mask coming from a saliency detection model or it has to be perfect foreground mask. Thanks!

Traceback (most recent call last): File "train.py", line 341, in train() File "train.py", line 313, in train checkpoint_callback = ModelCheckpoint(dirpath=os.path.join(args.basedir, args.expname), TypeError: init() got an unexpected keyword argument 'every_n_val_epochs'

Hi,

I'm running it on my own dataset using two GPUs and it stuck like this:

Any suggestions?

btw, here https://github.com/snap-research/NeROIC/tree/master/scripts it shouldn't be

cd utils/data_preproccess

It should be

cd scripts

instead.

Thank you for this great work!

I am wondering whether there are any model files like obj files generated during training procedure or test procedure. I just find some png files and mp4 files in logs and results directory respectively. I have executed all steps of Quick Start and Testing in READEME.md.

Your help is greatly appreciated!

In the process of running the code 'python train.py --config configs/milkbox_geometry.yaml --datadir ./data/milkbox_dataset', an error occurs, prompting 'AttributeError: module 'keras.backend' has no attribute 'is_tensor' ' Error, please, how do I fix this? Thanks!

Hello! I have been training the network on my own data and the rendering results are way worse than the ones from training on the provided datasets. I trained on a dataset that has 53 images. I kept the same parameters for the config files, changing the image size and name, of course.

One of the input images: And these are the result images taken from the logs folder: Visualizing the dense output from colmap:

What could be the possible reasons for such results?

I use the milkbox dataset to train the model and the training PSNR is about 29. However, the images synthesis in validation are nearly blank with a little noise, and the PSNR is extremely low. Is it overfitting?

I just run like this for the geometry model:

python train.py
--config
configs/milkbox_geometry.yaml
--datadir
data\milkbox_dataset