Hi there,

I'm trying to learn 3D scene with 360-degree view dataset like lego data in NeRF. It seems that the learning process is suffered from the initial error of SDF estimation. Of course, it is a different scenario with IDR dataset, however, the learning usually converges to local optimum Could you please let me know is there any strategy to learn the 360 view data properly?

Hello, I am trying to reproduce your results on the thin structures. I run the images through the COLMAP, calculate the camera_sphere.npz but the scale matrix is wrong. This is the scale matrix I get:

Scale matrix:
[[ 1.2317002e+03  0.0000000e+00  0.0000000e+00 -1.7486597e+02]
 [ 0.0000000e+00  1.2317002e+03  0.0000000e+00 -2.5129393e+01]
 [ 0.0000000e+00  0.0000000e+00  1.2317002e+03  1.0682924e+03]
 [ 0.0000000e+00  0.0000000e+00  0.0000000e+00  1.0000000e+00]]

I provide a visualisation of cameras in Unity3D. The cubes represent cameras, the rays from cubes outwards represent the camera direction where the blue rays are original cameras and the red rays are from the COLMAP. The XYZ cross represents world's center (0,0,0).

Visualisation of not scaled cameras:

Visualisation of scaled cameras: red cameras are scaled down way too much

However, it works on the DTU and BMVS datasets. Could you please provide any details on how did you get your camera_sphere.npz for the thin structures, or any idea what could be wrong or how to set the preprocess_cameras.py?

Hi, I can not reproduce the psnr you@Totoro97 mentioned in the supplementary materials with your pre-trained models.

Here are the psnr from pre-trained models:

Please tell me if I miss something.

Hello, I have been trying to do the following:

1. take a data, let's say bmvs_stone
2. feed images to the COLMAP
3. generate the cameras_sphere.npz
4. train NeuS on these data

To generate the cameras_sphere.npz I used this code from the IDR project. This code has many times resulted in error because it was not able to find points for normalisation, here. After rerunning the COLMAP a few times, I got 2 points and it was able to generate the scale_mat_xx. However, the results after training are blurry, not good.

I have tried using automatic reconstruction in COLMAP and also this script from the LLFF project.

To convert the output from the COLMAP to camera_sphere.npz I used this:

import numpy as np
import os
import colmap_read_model as read_model
from scipy.spatial.transform import Rotation as R


DIR = "/home/uriel/Downloads/DTU/dtu_scan24"
DST = os.path.join(DIR, "cameras_sphere.npz")


def load_colmap_data(realdir):
    camerasfile = os.path.join(realdir, 'sparse/0/cameras.bin')
    camdata = read_model.read_cameras_binary(camerasfile)

    # cam = camdata[camdata.keys()[0]]
    list_of_keys = list(camdata.keys())
    cam = camdata[list_of_keys[0]]

    h, w, f, cx, cy = cam.height, cam.width, cam.params[0], cam.params[1], cam.params[2]
    # w, h, f = factor * w, factor * h, factor * f
    hwf = np.array([h, w, f]).reshape([3, 1])
    k = np.array([[f, 0, cx],
                  [0, f, cy],
                  [0, 0, 1]])

    imagesfile = os.path.join(realdir, 'sparse/0/images.bin')
    imdata = read_model.read_images_binary(imagesfile)

    w2c_mats = []
    bottom = np.array([0, 0, 0, 1.]).reshape([1, 4])

    names = [imdata[k].name for k in imdata]
    print('Images #', len(names))
    perm = np.argsort(names)
    for i in imdata:
        im = imdata[i]
        R = im.qvec2rotmat()
        t = im.tvec.reshape([3, 1])
        m = np.concatenate([np.concatenate([R, t], 1), bottom], 0)
        w2c_mats.append(m)

    w2c_mats = np.stack(w2c_mats, 0)
    c2w_mats = np.linalg.inv(w2c_mats)

    return k, c2w_mats, perm


k, rts, perm = load_colmap_data(DIR)
cameras = {}
for i in perm:
    r = rts[i, :3, :3]
    t = rts[i, :3, 3]
    wm = np.eye(4)
    wm[:3,:3] = k @ r
    wm[:3,3] = k @ t
    cameras['world_mat_%d' % i] = wm
    cameras['scale_mat_%d' % i] = np.eye(4)

np.savez(DST, **cameras)

The code is inspired by this script.

I think the problem is that COLMAP uses a different coordinate system then NeuS and IDR, which I suppose is the same as OpenGL, where COLMAP says: The local camera coordinate system of an image is defined in a way that the X axis points to the right, the Y axis to the bottom, and the Z axis to the front as seen from the image.

Do you have any ideas how to solve it? Thanks

i use the default params ,preprocessed data using colmap script by you and train on my data,but get this strenge results:the target is good except the points around.

Hi. I can see that the scene bounds are calculated in this function: https://github.com/Totoro97/NeuS/blob/9dc9275d3a8c7266994a3b9cf9f36071621987dd/models/dataset.py#L160-L166

But I am not sure how scene bounds can be estimated just from ray origins and directions. Can you please explain what these different operations are doing conceptually ?

I want to put texture to ply files produced by pretrained model of bmvs_bear; what I need is: A:ply x y z and face ,the point in ply should in world coordinate. and i get it in this way:（i check the code ,this produce a set of points in world coordinates
validate_mesh(self, world_space=True, resolution=512, threshold=0.0)） python exp_runner.py --mode validate_mesh --conf <config_file> --case <case_name> --is_continue # use latest checkpoint all is the same with the example not change. B: instinct and extrinct of per pic and i get it from Dataset class by adding a member function def save_cam(self): print("start to save!") intrinsics_ = self.intrinsics_all.cpu().numpy() pose_all_ = self.pose_all.cpu().numpy() max_h_w = max(self.W ,self.H) for index in range(self.n_images): print ('images_path : %s' % self.images_lis[index]) current_cam_path = self.images_lis[index] current_cam_path = current_cam_path.replace('png','cam') print (type(current_cam_path)) print ('cam_path : %s' % current_cam_path) fx = intrinsics_[index][0][0] cx = self.W - intrinsics_[index][0][2] cy = intrinsics_[index][1][2] d0 = 0.0 d1 = 0.0 paspect = self.W * 1.0/self.H ppx = cx/self.W ppy = cy/self.H print ('fx : %f cx : %f cy : %f' % (fx,cx,cy)) print ('intrinsics_ : %s' % intrinsics_[index]) tx = -pose_all_[index][0][3] ty = pose_all_[index][1][3] tz = -pose_all_[index][2][3] R00 = -pose_all_[index][0][0] R01 = -pose_all_[index][0][1] R02 = -pose_all_[index][0][2] R10 = pose_all_[index][1][0] R11 = pose_all_[index][1][1] R12 = pose_all_[index][1][2] R20 = -pose_all_[index][2][0] R21 = -pose_all_[index][2][1] R22 = -pose_all_[index][2][2] file = open(current_cam_path, 'w')#negetive here to trans points in neus to i need file.write('%f %f %f %f %f %f %f %f %f %f %f %f\n'%(tx,ty,tz,R00,R01,R02,R10,R11,R12,R20,R21,R22)) file.write('%f %f %f %f %f %f\n'%(fx/max_h_w,0.0,0.0,paspect,ppx,ppy))

unfortunately,i got a bad result,so there must be a error in this pipeline.would you like to help me to ensure A and B()，to see where is error? plus：the pic indicates the coordinate In neus and I need

Hi, I would like to ask the derivation process of equation 6. Especially moving cosine outside the integration: As stated in the paper, here we treat cosine as a constant, and I know that the constant factor in integral can be move outside the integral, but why we end up move |cos(theta)|^{-1} out instead of -|cos(theta)| ? TIA.

Hi,

Thanks for your excellent work. I'm curious about the MAE of SDF values mentioned in sec 4.3. How do you get the GT SDF? Do you calculate it by querying the nearest point with the provided mesh or pointcloud in DTU? Could you please share the evaluation code for this metric in convenience?

after trained without mask, I found some extra meshes such as white planes, how can I evaluate the chamfer distance between the pred mesh and gt point cloud. If I sample many points in the surface of mesh, there will be many points in white planes in the same time.

Thanks for the great work!

I want to draw the SDF contour lines on the cutting plane. How do you draw the pictures like Figure 13. in your paper?

Thanks a lot!

Dear authors,

Thanks a lot for your great work! I have tried to use the lego data to train and the results seems really bad as the attached image shows, do you have any idea to debug it? The training data and conf file is downloaded from issue #9 .

Thank you very much!

Best, Yuchen

Hi, I appreciate to your awesome work!

I have three questions.

1. How to train NeuS on forward-facing scene(e.g. fern scene in NeRF dataset) using dataloader for llff formatted scene? I used dataloader in official implementation of NeRF to load the offered dataset from them. In their dataset,poses_bounded.npz, which is already colmap-preprocessed data, are exists and I used it. Hence, near and far bound are set following the npz file and they are not estimated as unit-sphere as you implemented for NeuS. In addition, we use the camera poses of the scene from poses_bounded.npz as it is and not rescale as you did in your code following original NeRF setting. Is there any problem for this setting to train the NeuS on real forward-facing scene? I wonder that we have to rescale the forward-facing scene to be in unit-sphere scale and rescale the camera pose for training NeuS.

2. How to set the object bounding box parameters(object_bbox_min, object_bbox_max in dataset.py) for forward-facing scene? I guess it would not necessary for forward-facing scene since they are scene-level NeRF framework and not object-centric scene. However, I'm confused how to set the above values to generate mesh from trained NeuS for forward-facing scene.

3. Related to question 2. Unlike the dataset used to train NeuS on this repo, forward-facing scene usually include sky-like part which are regarded as infinite depth. Hence I wonder it is necessary to use the NDC coordinate to train the NeuS on forward-facing scene. In addition, if it is necessary to set the object bounding box parameters to exclude the sky-like part when we generate mesh from trained NeuS on forward-facing scene, how to set the values?

Thank you for your work again and I will wait for your reply!

I haved trained the model using my pictures and womask.conf but can not get any result. A part of the output text:

[export.py:72 -          export_mesh() ] Exporting 0 faces as PLY
[exp_runner.py:342 -        validate_mesh() ] End

When validating meshes,there is always nothing to export . I don't know what happends The exported .ply files are also empty

Dear authors, thank you for the amazing work! I am a little confused about the code implementation about how to plot the normal map in exp_runner.py

normal_img = np.concatenate(out_normal_fine, axis=0)
rot = np.linalg.inv(self.dataset.pose_all[idx, :3, :3].detach().cpu().numpy())
normal_img = (np.matmul(rot[None, :, :], normal_img[:, :, None])
              .reshape([H, W, 3, -1]) * 128 + 128).clip(0, 255)

why do we need to put the c2w rotation matrix upon the normal map? Sorry, I am a freshman in this field and have little understanding of the graphics, and I didn't see the code in VolSDF or other works, they just multiply 255 and save it in images. could you tell me what's the difference between these methods concerning how to plot normal maps? Thanks again

I keep getting IndexError: index 3678 is out of bounds for dimension 1 with size 3024 when training my models. May I know what I am doing wrong?

Hello Wooden Load data: Begin Load data: End 0%| | 0/300000 [00:00<?, ?it/s] Traceback (most recent call last): File "exp_runner.py", line 392, in runner.train() File "exp_runner.py", line 105, in train data = self.dataset.gen_random_rays_at(image_perm[self.iter_step % len(image_perm)], self.batch_size) File "/home/user/NeuS/models/dataset.py", line 119, in gen_random_rays_at mask = self.masks[img_idx][(pixels_y, pixels_x)] # batch_size, 3 IndexError: index 3678 is out of bounds for dimension 1 with size 3024