Dear author,

From the output of your code, the spatial size of MPI_b is 400 by 400, which is different with the spatial size of other output.

So does your program need interpolation process when rendering on web server?

Why the MPI_b needs to be downsampled during saving, but using same size when training?

Thanks

Hello and thanks for sharing this nice work! I 've been trying to make better outputs with nex but the only thing I got was low-quality output.

I trained 2k-images with this code.

python train.py -scene ${PATH_TO_SCENE} -model_dir ${MODEL_TO_SAVE_CHECKPOINT} -http -layers 12 -sublayers 6 -hidden 256

but I got output-images like these. (the images which are automatically created in video_output directory after training )

Even I created a dataset following the images collection method shown in LLFF and successfully installed Colmap and other requirements as well, I have no idea what's wrong with it.

If there is any other way to get better outputs, please help me out. :)

and my environments are like those below. RTX-2080Ti Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz Ubuntu 18.04. CUDA Toolkit 10.2

Thanks

For real-time rendering, is it true that H_i(v) is first offline-computed on the reference view v (pre-defined), and then warped to new views (unknown)? which mean there is no network inference in the real-time rendering stage.

is it possible to share the code of the online viewer (webgl)?

Hi,

Thanks for sharing this great work! I'm confused about the pose convention of the following lines in def nerf_pose_to_ours(*). Could you explain the geometric meaning behind it?

https://github.com/nex-mpi/nex-code/blob/eeff38c712ac9a665f09d7c2a3fdf48ae83f4693/utils/sfm_utils.py#L323-L325

Hi @pureexe, thanks for your great work, but when I trained on m own datasets for a while, I got a Cuda error, when I change to train on only one GPU, it can train for a longer time, but can also trigger this error

the error messages like below: train.py in forward cof=pt.repeat_interleave(cof,args.sublayers,0) runtimeerror: an illegal memory access was encountered

But when I use the demo-room datasets to train, it seems the training phase is normal. I use the colmap to preprocess the datasets and get the hwf_cxxy.txt and poses_bounds.npy using the scripts you provided.

btw, when train on my own datasets, how to set the plane.txt? hope you can give some advice, thanks~

All 3d points along one ray have the same viewing direction. So when rendering, isn't it enough to input only one viewing direction rather than input all duplicate viewing directions into the basis function?

Below is the result I checked by myself. out2 is the basis function value.

As you can see, all 32 values have the same value of 0.1176. Since the input is the same, the output is of course the same. My question is, do I really need to waste network memory? Instead of having 32 inputs, isn't it enough to have just 1 input?

Hi, thanks for your great work!

I found there are some black lines in the boundaries of some images, in the Shiny dataset (for example, CD):

After I resize the image to the target width (1008), the black line still exists:

Could you help figure out the reason behind this issue? I would like to know if I should remove the boundary pixels during training (and also in testing).

Thank you very much!

After Download the shiny datasets through one-drive link, I can't decompress the zip file.

To extract the file, I repaired the compressed file through following command.

zip -FF my_zip --out my_zip_ver2.zip

But, the file still have some problems after repairing.

When I decompress the repaired file, there is the log file which notice the error logs.

• many files does not exist.

1062 files does not exist.

Is there any problem with the uploaded file?

Or how to download and extract the complete dataset?

I tried download on OSX and Ubuntu, but both failed.

Do I have to use window to download the file?

when follow the steps

conda env create -f environment.yml
./download_demo_data.sh
conda activate nex
python train.py -scene data/crest_demo -model_dir crest -http

Here comes the problem, I don't know what happens.

"train.py" in <module>
  751:  train()
  train.py
"train.py" in train
  633:  output = model(dataset.sfm, feature, output_shape, sel)
  train.py
"module.py" in _call_impl
  889:  result = self.forward(*input, **kwargs)
"train.py" in forward
  334:  warp, ref_coords = computeHomoWarp(sfm,
  train.py
"train.py" in computeHomoWarp
  158:  prod = coords @ pt.transpose(Hs, 1, 2).cuda()
  train.py
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`

Hey,

I'm able to open the demos you generated on my mobile phone. however, I'm not able to see my own generated MPIs on my mobile. They simply do not show up, I only see the slider of layers. I'm hosted my MPI on the web, and used the same viewer you use for mobile. textures are all loaded, but then it does not show anything. Do you I need to do any further modification on the MPI in order to view it on mobile web ? Thanks for your support and great work

Firas

Thank you sharing this amazing work. If you have them, could you share the rendered test images for the trex scene like you have for the other scenes here : https://drive.google.com/corp/drive/folders/1OLSy326rxCKMYRo4K7S27ew9D8NzfJo4

According to the paper, on the Real Forward-Facing dataset (Table B.1), PSNR is 25db on average, ranging between 20-32db. On the Spaces dataset (Table B.5), PSNR is stably around 35db or higher for each scene.

Do you have any insights why that's happening?

My Setup

• I'm using colab.
• I'm using my own set of images.
• I didnt tweek anything after colmap.

My Problem

When training, it cant load data since util/load_llff.py:107 return None after mismatch warning. This is the screenshot:

Another Problem

I tried to remove the mismatch images and eventually the colmap give following error: FileNotFoundError: [Errno 2] No such file or directory: 'data/demo/dense/sparse/cameras.bin' Here is the screenshot of log: I wonder what do these mean and how to fix these problem. If more infomation is need, just let me know. Any help would be greatly appreciated.

How can I check the given data loader and data format for Shiny dataset ?

In addition, what camera coordinate Shiny dataset use?

For example, x,y,z is (right, up, backward) in NeRF.

Can I use the LLFF loader for Shiny dataset as well?

1. I was curious about the nerf_pose_to_ours function, and I read the article below. But there are still some things I don't understand. https://github.com/nex-mpi/nex-code/issues/13

As I understand the Pose value is the Camera to world matrix, and each column represents the x-axis, y-axis, z-axis, and location of the camera in the world coordinate system.

If I want to change the coordinate axis of the camera from the opengl coordinate system (right, up, backward) to the opencv coordinate system (right, down, forward), the pose values [r1, r2, r3, t] are set to [r1, -r2, -r3, t], isn't it? Why does Translation change? Isn't the world coordinate system fixed?

The poses_bounds.npy file stores the camera coordinate axes as (down, right, backward). When you change this from NeRF to OpenGL coordinate system (right, up, backward), don't you do it like the following? Why is the method of changing the coordinate axes in the nerf_to_our_pose function different from the method of changing the coordinate axes below?

https://github.com/nex-mpi/nex-code/blob/eeff38c712ac9a665f09d7c2a3fdf48ae83f4693/utils/load_llff.py#L245-L254

Doesn't the world coordinate system matter whether it is opencv convention or opengl convention? Isn't the world coordinate system determined independently? Maybe it's because the nerf_to_our_pose function is located after recenter..? I'm confused.

2. It makes sense to multiply the focal length by the scale factor when resizing the image. By the way, why add 0.5 to the principal point, multiply, and subtract 0.5 again? Can't we just multiply by sw? I'm curious about the hidden meaning here.

https://github.com/nex-mpi/nex-code/blob/eeff38c712ac9a665f09d7c2a3fdf48ae83f4693/utils/sfm_utils.py#L188-L191

Are the sampled x,y,d already warped before feeding them into the network? Cause in the algorithm in the paper warping is computed after the color info is regressed?