In order to run IDR on new data [DIR PATH], you need to supply image and mask directories, as well as cameras.npz file containing the appropriate camera projection matrices.

Dear authors,

Do we need to convert camera projection matrices in DTU dataset from txt to npy files?

Thanks

Hi,

Thank you for sharing the code. I trained the model. When I ran eval.py to extract the mesh using the latest checkpoint I am getting Cuda OOM in the File "../code/utils/plots.py", line 197, in get_surface_high_res_mesh which is the line grid_points = torch.cat(g, dim=0) I tried to change the following code snippet in plots.py, but it did not help. My GPU has 11GB memory. I also tried using 2 gpus, but I think the eval code uses only a single gpu. I also tried torch.cuda.empty_cache() to clear the cuda cache but I still get the OOM error. Could you please provide some guidance on how to fix the OOM problem? Thanks.

g = []
for i, pnts in enumerate(torch.split(grid_points, 100000, dim=0)):
    g.append(torch.bmm(vecs.unsqueeze(0).repeat(pnts.shape[0], 1, 1).transpose(1, 2),
                       pnts.unsqueeze(-1)).squeeze() + s_mean)
grid_points = torch.cat(g, dim=0)

Hi there,

First of all congrats on the project and on the code. It's really neat 🔥 .

I'm trying to run the code with my own scene and, even though the losses are printed with kind of meaningful values, they do not decrease and the learnt scene doesn't resemble at all at what it should. I was wondering if the issue could be related to the coordinate system used.

My coordinate system is this way:

• Cameras look towards z.
• Usually, if the object is at the origin, cameras position are somewhere in -z.

Could it be you are assuming an OpenGL-like coordinate system and that it could be the reason it doesn't work?

• Here cameras look towards -z.
• Cameras are usually located around +z if the object is at the origin.

Any thoughts will be very welcome.

Thanks in advance!

Hello, I am a big fan of this project I wonder how you get the scale matrix with DTU dataset. If you don't mind, could you share the code which you use to get scale matrix?

I think I solved it Thank you

I did try to run the last instruction , where for GEOMETRY_ID , I have given the trained surface model with surface_world_coordinates.ply and same for appearance_id too, even I have changed those generated evals to code and then tried to run the code by specifying the location_path saying evals/dtu_trained_cameras/surface_world_coordinates.ply

But nothing worked for me, can you be bit more clear about the instructions for the execution. Thank you

Hi,

I have a quick question regarding the design choice made in this work. In the rendering network, positional encoding is applied to view directions but not positions. This is quite different from NeRF, where they positional-encode the 3d positions. I just wonder if there's any special consideration behind this choice? Thank you.

Hi @lioryariv, this is great work!

I noticed that the published code doesn't contain scripts for quantitatively evaluating the reconstructed surface against the DTU ground truth. Your packaged DTU dataset doesn't contain ground-truth shapes either.

Could you please share your scripts for quantitative evaluation of DTU scenes? If you can also share the ground truth shapes (packaged with your DTU data) to use with your scripts, that would be amazing!

Thanks and Regards, Shubham

I noticed that the cameras.npz file provided for the DTU dataset has some additional matrices called "camera_mat_{i}". Are these the 3x3 calibration matrices K from P = K[R | t]?

I tried to extract K, R and t from P (which are "world_mat_{i}") using opencv: K, R, t, _, _, _ = cv2.decomposeProjectionMatrix(P2)

The K that I get from here does not match with the corresponding "camera_mat_{i}". Also the t values that I get are confusing to me. t has the shape (4, 1) and usually the 4th entry is always 1, but that is not happening here.

UPDATE: The 't' values were actually okay. I forgot to divide it by the 4th entry

In the code: https://github.com/lioryariv/idr/blob/44959e7aac267775e63552d8aac6c2e9f2918cca/code/model/implicit_differentiable_renderer.py#L249 I found that the normal predicted by the implicit network is directly input to the rendering network without normalization.

I want to know if it is well-designed by you, and why?

Hey there,

This is not an issue, just a question. Using softplus activation is significantly slower than using for instance ReLU. Is there any technical (theoretical or practical) reason why you chose softplus in front of other more efficient activations?

Thank you!

In the data_convention.md file, the preprocess script says that it generates the needed cameras.npz file. The code loads an initial cameras.npz file, but how should we build this cameras.npz file to run the preprocess code on new images ?

Hi, thanks for your great work. I am reading your codes and one thing confuses me. For the spherical tracing part, accoring to the paper, it should find the first intersection point (which is also nearer) first and if it is not convergent, the algorithm will begin at the another intersection point, which is the farthest point. However, when I read the codes, the start_point seems to be farther from the camera center than the end_points because the start_points choose the index 0 of "sphere_intersections_points" and the end_points choose index 1. Because according to "get_sphere_intersection" function, the index 0 stores (-under_sqrt - ray_cam_dot) and index 1 stores (under_sqrt - ray_cam_dot), which means the tensors on index 0 have larger absolute value. So it seems that start_points will be farther than the end_points, which is comfusing. Did I get anything wrong?

Hi folks,

Thanks for this awesome work. I'm wondering how to render the "spiral" videos demonstrated in the project page?

Any reply would be appreciated!

Best, Peihao

Thanks for the great work. For my side, I am trying to train one appearance and transfer it to other unseen geometry objects, but the results are not so correct, may I know is there anyone else also encountered this issue? Thanks in advance.

Thanks for the great work. I was trying to run it on some images captured through my camera. I used the colmap to get the world matrices. But it is giving the following error while trying to run preprocess_camera.py

preprocess_cameras.py:87: DeprecationWarning:np.floatis a deprecated alias for the builtinfloat. To silence this warning, usefloatby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.float64here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations mask_points_all.append(np.stack((xs,ys,np.ones_like(xs))).astype(np.float)) preprocess_cameras.py:45: DeprecationWarning:np.floatis a deprecated alias for the builtinfloat. To silence this warning, usefloatby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.float64here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations cur_l_1=Fj0 @ np.array([curx,cury,1.0]).astype(np.float) Number of points:0 preprocess_cameras.py:169: RuntimeWarning: Mean of empty slice. centroid = np.array(all_Xs).mean(axis=0) /home/kshitiz/miniconda3/envs/idr/lib/python3.7/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) /home/kshitiz/miniconda3/envs/idr/lib/python3.7/site-packages/numpy/core/_methods.py:263: RuntimeWarning: Degrees of freedom <= 0 for slice keepdims=keepdims, where=where) /home/kshitiz/miniconda3/envs/idr/lib/python3.7/site-packages/numpy/core/_methods.py:223: RuntimeWarning: invalid value encountered in true_divide subok=False) /home/kshitiz/miniconda3/envs/idr/lib/python3.7/site-packages/numpy/core/_methods.py:254: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) Traceback (most recent call last): File "preprocess_cameras.py", line 244, in <module> get_normalization(opt.source_dir, opt.use_linear_init) File "preprocess_cameras.py", line 207, in get_normalization normalization,all_Xs=get_normalization_function(Ps, mask_points_all, number_of_normalization_points, number_of_cameras,masks_all) File "preprocess_cameras.py", line 175, in get_normalization_function centroid,scale,all_Xs = refine_visual_hull(masks_all, Ps, scale, centroid) File "preprocess_cameras.py", line 101, in refine_visual_hull points = points + center[:, np.newaxis] IndexError: invalid index to scalar variable.

Hi! Thank you for the great work, I'm trying to get system to work on my own images, but am facing some difficulties. I obtain the position information using an aruco board on which I place my object. Now, the algorithm moves past the ray-tracing step properly, but then the algorithm outputs an rgb-loss of "0.0". Did anyone encounter something similar? I am very grateful for any type of response :)