I've been able to run this on my own data (slowly, on a 3070) using U^2Net matting and colmap2nerf code from instant-ngp (if others are following this, remove the image extensions, and it should run on windows) The issue I'm having is val, and test data or the lack of it. Are there config options to remove the requirement for those datasets. Copying and renaming the json lists to val and train works, but is rather cumbersome, and I was wondering if I was missing a better option with the NeRD dataset preparation, which use manual masks not applied to the image itself (which is possible with U^2 net, and would help with colmap mapping since the background can still be used for feature tracking.)

I haven't looked into what data is required to be presented in the config, nor resolution/training options, but I'm just wondering what's the generally intended arguments and method for presenting your own data, when no val is available.

I am trying to get nvdiffrec to do light estimation in a controlled setting. I suspect that there are some coordinate frame and lighting issues. Here is the procedure I am following: First, I generate a synthetic dataset using a custom environment map from the NeRF lego script. Then, I try to train the model from the generated dataset without giving an environment map, or a reference mesh. I end up with a mesh that is flipped 90 degrees on the x-axis and the lighting is badly estimated.

Here is the result mesh and the lego before training:

Thank you for the great work! I followed the one time setup on ubuntu 18.04. Python=3.9, CUDA=11.3, PyTorch=1.10. Then I had a trial on nerd_ehead by running: python train.py --config configs/nerd_ehead.json In dmtet_mesh folder, the mesh looks good : But in mesh folder, the mesh.obj looks confusing: I did not change any code in this repo so I did not know why this happened. Could you please help me solve this problem? Thank you for your time!

Hello, thank you for your great work. In #9 and in Paper 8.5, You expressed you used DVR based MLP to optimize SDF. And I have several question about it.

1. Can you tell me how many iter you have run to pre-train sphere?
2. Did you set the MLP output shape as (size, sdf) which mean (size,1) or (size, sdf+deform) which mean (size, 4) ?
3. Did you use Modulelist in DVR decoder to process latent condition code c too?

Hello. First of all, thank you very much for your excellent work. I'm sorry I'm not very familiar with graphics engines, but now I want to render the final output, such as hotdog's Mesh. There are some questions below hope to get your answer: 1.According to the description in the article, kd.png\ks.png\kn.png describes the material of the model，But the final output has.mtL file, is.mtL equivalent to three PNG files？Now when I use mesh.load_mesh(mesh.obj), the api will directly use mesh.mtl instead of loading three Png files 2. If.mtL means the same as three Png files, I can render by loading the model and loading Png files?

Thank you for the amazing work!!

I did a quick trial on the custom in-the-wild data and found that there are some noise in fitting results. Did I mess up something? Have you observed this phonenmon before? Note that I fixed the vertex for mesh and just optimized the lighting / material.

| rendered | reference | |---|---| | | | | | |

The texture maps looks like: | kd | ks | normal | |---|---|---| | | | |

Looking forward to your great help! Thanks a lot

Hello,

I have run "python train.py --config configs/nerf_lego.json" with default setting. The picture below shows my mesh output.

1. The smooth surface is not flat (as shown in the red box). How should I improve it?
2. Lots of details are loss due to the geometric simplification. How can I get a model with more details?

Thank you in advance!

Hi, I have been stuck on this error for some time now, the optimize_mesh function is not functioning properly on CUDA I'm running a custom dataset with colmap transforms matrix generated from colmap2nerf script of instant-ngp.

Any help is appreciated

Resources used :

• torch 1.10.2+cu113
• cudatoolkit 11.3.1
• tinycudann 1.6

---------
config configs/nerf_shoe.json
iter 5000
batch 8
spp 1
layers 1
train_res [800, 800]
display_res [800, 800]
texture_res [2048, 2048]
display_interval 0
save_interval 100
learning_rate [0.03, 0.01]
min_roughness 0.08
custom_mip False
random_textures True
background white
loss logl1
out_dir out/nerf_shoe
ref_mesh data/shoe-2-nvdiff
base_mesh None
validate True
mtl_override None
dmtet_grid 128
mesh_scale 2.1
env_scale 1.0
envmap None
display [{'latlong': True}, {'bsdf': 'kd'}, {'bsdf': 'ks'}, {'bsdf': 'normal'}]
camera_space_light False
lock_light False
lock_pos False
sdf_regularizer 0.2
laplace relative
laplace_scale 3000
pre_load True
kd_min [0.0, 0.0, 0.0, 0.0]
kd_max [1.0, 1.0, 1.0, 1.0]
ks_min [0, 0.08, 0.0]
ks_max [1.0, 1.0, 1.0]
nrm_min [-1.0, -1.0, 0.0]
nrm_max [1.0, 1.0, 1.0]
cam_near_far [0.1, 1000.0]
learn_light True
local_rank 0
multi_gpu False
---------
NERF dataset path:  data/shoe-2-nvdiff/transforms_train.json
DatasetNERF: 143 images with shape [3024, 4032]
DatasetNERF: 143 images with shape [3024, 4032]
Encoder output: 32 dims
Traceback (most recent call last):
  File "train.py", line 595, in <module>
    geometry, mat = optimize_mesh(glctx, geometry, mat, lgt, dataset_train, dataset_validate, 
  File "train.py", line 384, in optimize_mesh
    target = prepare_batch(target, 'random')
  File "/opt/users/saptarshi.majumder/tmp/miniconda3/envs/nvdiffrec/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "train.py", line 89, in prepare_batch
    target['img'] = torch.cat((torch.lerp(background, target['img'][..., 0:3], target['img'][..., 3:4]), target['img'][..., 3:4]), dim=-1)
RuntimeError: The size of tensor a (3) must match the size of tensor b (0) at non-singleton dimension 3```

Hi, sorry to ask about this again, I know #3 and some other issues are about this problem, but it seems that there only are ways of getting poses for the images are discussed, and same for the 'https://github.com/bmild/nerf#generating-poses-for-your-own-scenes' you mentioned in another issue before, those methods truly help a lot, thanks for you and those who contributed! Now I am confused with the masks getting process, i find there are two different types of masks in your examples, in nerd dataset there's a 'masks' directory containing mask images corresponding with the ones in 'images', and for the nerf synthetic dataset, the 4-th channel image contains the masks information in the image itself as you said.

So I guess my biggest question is that how to get the mask images like the first way or how to get 32bit images with alpha mask. Besides, i want to know if this is true, that seperate mask images need to be used with .npy file, and alpha-mask type needs to be used with .json file. Please correct me if i get something wrong about all these. Thanks for your work and your time.

Hi,

Is there any limitation on the training image resolution?

I used just 12 images of resolution 4640x3472, and got this error though I successfully generated mesh for custom images with lower resolution. I am using Nvidia RTX A6000 with 50GB, so memory shouldn't be an issue.

Are there some parameters in the config that I can tweak to occupy less memory with compromising mesh quality?

What parts of the input in the training phase are the multi-view images, camera poses and masks corresponding to multi-views? Will a .obj file be generated after training as a model for testing? What are the parts of the input in the testing phase? Thank you very much for your reply, urgent!

Hi, again, thanks for your great work, I've noticed that in your code that the environment map can be loaded by 'lgt = light.load_env' rather than create a trainable one. And by your design idea, it needs the flag 'learn_light' to be false to do this, consequently, the optimize process won't try to optimize the light because the FLAGS.learn_light is false. I just wonder that what if I still want to optimize the light I loaded in the beginning(to do this only needs to change the code a little bit), I just want to know if is this a wrong idea to optimize a already loaded environment light.(for example I can pre-capture the envionment around the object and get a hdr environment map). Could you share some insights about this? THANKS!

...It saves a manual step later when cloning/pulling :)

Example:

git add script.sh
git update-index --chmod=+x script.sh
git commit -m "Script is now executable by default"

Hi when I tried to run the nerf-synthetic dataset with configs/nerf_ficus.json, it shows the following failure info after finishing the first stage optimization:

Writing material:  out/nerf_lego/dmtet_mesh/mesh.mtl
Done exporting mesh
Traceback (most recent call last):
  File "train.py", line 622, in <module>
    optimize_geometry=not FLAGS.lock_pos)
  File "train.py", line 415, in optimize_mesh
    img_loss, reg_loss = trainer(target, it)
  File "/home/sarahwei/anaconda3/envs/dmodel/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "train.py", line 299, in forward
    return self.geometry.tick(glctx, target, self.light, self.material, self.image_loss_fn, it)
  File "/home/sarahwei/code/nvdiffrec/geometry/dlmesh.py", line 56, in tick
    buffers = self.render(glctx, target, lgt, opt_material)
  File "/home/sarahwei/code/nvdiffrec/geometry/dlmesh.py", line 49, in render
    num_layers=self.FLAGS.layers, msaa=True, background=target['background'], bsdf=bsdf)
  File "/home/sarahwei/code/nvdiffrec/render/render.py", line 230, in render_mesh
    rast, db = peeler.rasterize_next_layer()
  File "/home/sarahwei/anaconda3/envs/dmodel/lib/python3.6/site-packages/nvdiffrast/torch/ops.py", line 378, in rasterize_next_layer
    result = _rasterize_func.apply(self.raster_ctx, self.pos, self.tri, self.resolution, self.ranges, self.grad_db, self.peeling_idx)
  File "/home/sarahwei/anaconda3/envs/dmodel/lib/python3.6/site-packages/nvdiffrast/torch/ops.py", line 246, in forward
    out, out_db = _get_plugin(gl=True).rasterize_fwd_gl(raster_ctx.cpp_wrapper, pos, tri, resolution, ranges, peeling_idx)
RuntimeError: Cuda error: 700[cudaMemcpy3DAsync(&p, stream);]
Aborted (core dumped)

Hi, thank you for the excellent work! But I found self-intersections in the generated mesh, and I wonder why the mesh generated by marching tet is not manifold.

Hi, I found that resizing images and then training makes the convergence much slower as compared to keeping the original resolutions, but for higher resolutions i get CUDA OOM errors. Is there a tradeoff that hits the sweet spot or is there an alternative for this?

Any help is appreciated

hello，Thank you for sharing your great work！In the previous issues, you said that this parameter “mesh_scale” represents the size of the tetrahedron。when I test on other dataset（human public dataset）, i find some trouble。when i set “mesh_scale” as 3.0,the result as follow when i set “mesh_scale” as 10 ，the result as follow . The results are from pass1. I don't understand the influence of this parameter on reconstruction. Why is it completely white when the parameter is small? And in second image, could you have any good suggestions on how to prevent some areas from being cut？I want to find out the influence of Hyperparameter on training（in .json file) Looking forward to your reply!