Congrats on your great work and thank you for releasing the code. I am curious to train my custom dataset but get stuck in the dataset preparation process. I also notice that you mention in the paper: "We refine the initial GPS-derived camera poses in the Mill 19 and UrbanScene3D datasets and the estimates provided in the Quad 6k dataset using PixSFM."

I have the following questions:

1. How are the initial camera poses derived from drone GPS and IMU sensors? How are the initial GPS-derived camera poses refined using PixSFM? Could you kindly provide the script that can generate camera poses?
2. Can the camera poses estimated by COLMAP work with Mega-NeRF? Or should also be refined by PixSFM?

Thanks in advance and really looking forward to your reply! Xingyi

I read the paper about mega-nerf, which is amazing. Could I ask some questions like the arguments of the code? For example, what is the mean of '--chunk_paths $SCRATCH_PATH '? I do not see the notes of this parameter.

Hi, just to confirm the format of c2w matrix. Is it c2w[:3,:3] corresponds to rot matrix in order [x,y,z] and c2w[:3,-1] the camera position in order [z,x,y]? I noticed in _truncate_with_plane_intersection you use rays_o[:, :, 0] < altitude.

Hi, thank you for your great open source and quick response to my previous questions. I am trying to train mega-nerf to reproduce the results. I have a question related to the validation images. As in the code below: https://github.com/cmusatyalab/mega-nerf/blob/306e06cc316dd4f5c84d0610308bcbc208228fc3/mega_nerf/runner.py#L620 It seems that the validation images are also used in training. To my understanding, it is because we need to learn the appearance embeddings of the validation images. I would to like confirm this setting and want to ask whether I understand this setting correctly. Thank you very much.

Best regards, Zhenxing Mi

thank you for your excellent work! I have a question. I use dataset building and after training i cannot see the results of whole scene. Is there a .mp4 or some images exist to show the training result? Besides: I notice in render_images.py ,which requires input including pose.txt/intrinsics.txt/embeddings.txt. how can i get these ?

Hi, seems the camera parameter link of Campus has broken (https://storage.cmusatyalab.org/mega-nerf-data/campus-pixsfm.tgz). Could you please update the link so we could fully test on the datasets? Thank you very much.

Hi I used the quad6 dataset and set the grid_dim to 10, 10

However, I get the following cuda_outof_memory error: RuntimeError: CUDA out of memory. Tried to allocate 18.31 GiB (GPU 0; 23.70 GiB total capacity; 18.57 GiB already allocated; 3.13 GiB free; 18.58 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I'm using 3090 with 8 cards. I dont understand why it caused the out of memory issue.

Weide

I have trained the building dataset with 1000+ images and a sample of 50 images of the dataset. I trained on 4 3090 GPUs and it took almost 40 hours. However, the train time of the small dataset was almost as same as big one. I was wondering how to speed up this process. Thanks a lot.

I get this error after a while trying to train (on multiple nodes with multiple gpus each) a model on the building dataset, for chunk/submodule 0 (using the provided cluster masks) Any tips on debugging this? Seems like somewhere there is an issue in a sample.

 28%|██▊       | 140533/500000 [15:32:21<45:35:06,  2.19it/s]
 28%|██▊       | 140534/500000 [15:32:22<50:19:14,  1.98it/s]
 28%|██▊       | 140535/500000 [15:32:23<53:45:04,  1.86it/s]
 28%|██▊       | 140536/500000 [15:32:23<44:03:00,  2.27it/s]
 28%|██▊       | 140537/500000 [15:32:23<49:43:21,  2.01it/s]
 28%|██▊       | 140538/500000 [15:32:24<55:23:35,  1.80it/s]
 28%|██▊       | 140539/500000 [15:32:24<46:40:38,  2.14it/s]
 28%|██▊       | 140540/500000 [15:32:25<46:16:18,  2.16it/s]
 28%|██▊       | 140541/500000 [15:32:25<38:48:16,  2.57it/s]
 28%|██▊       | 140542/500000 [15:32:25<34:51:35,  2.86it/s]
 28%|██▊       | 140543/500000 [15:32:26<38:54:21,  2.57it/s]
 28%|██▊       | 140544/500000 [15:32:26<46:17:54,  2.16it/s]
 28%|██▊       | 140545/500000 [15:32:27<40:27:56,  2.47it/s]
 28%|██▊       | 140546/500000 [15:32:27<46:56:04,  2.13it/s]
 28%|██▊       | 140547/500000 [15:32:28<52:37:22,  1.90it/s]ERROR:torch.distributed.elastic.multiprocessing.errors.error_handler:{
[stderr]  "message": {
[stderr]    "message": "ValueError: cannot reshape array of size 42028165 into shape (42107792,)",
[stderr]    "extraInfo": {
[stderr]      "py_callstack": "Traceback (most recent call last):\n  File \"/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py\", line 345, in wrapper\n    return f(*args, **kwargs)\n  File \"/tmp/code/mega_nerf/train.py\", line 24, in main\n    Runner(hparams).train()\n  File \"/tmp/code/mega_nerf/runner.py\", line 226, in train\n    dataset.load_chunk()\n  File \"/tmp/code/mega_nerf/datasets/filesystem_dataset.py\", line 80, in load_chunk\n    chosen, self._loaded_rgbs, self._loaded_rays, self._loaded_image_indices = self._chunk_future.result()\n  File \"/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/concurrent/futures/_base.py\", line 438, in result\n    return self.__get_result()\n  File \"/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/concurrent/futures/_base.py\", line 390, in __get_result\n    raise self._exception\n  File \"/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/concurrent/futures/thread.py\", line 52, in run\n    result = self.fn(*self.args, **self.kwargs)\n  File \"/tmp/code/mega_nerf/datasets/filesystem_dataset.py\", line 111, in _load_chunk_inner\n    loaded_pixel_indices = torch.IntTensor(np.load(self._ray_arrays[next_index]))\n  File \"/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/site-packages/numpy/lib/npyio.py\", line 440, in load\n    return format.read_array(fid, allow_pickle=allow_pickle,\n  File \"/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/site-packages/numpy/lib/format.py\", line 787, in read_array\n    array.shape = shape\nValueError: cannot reshape array of size 42028165 into shape (42107792,)\n",
[stderr]      "timestamp": "1651655215"
[stderr]    }
[stderr]  }
[stderr]}
[stderr]Traceback (most recent call last):
[stderr]  File "/tmp/code/mega_nerf/train.py", line 28, in <module>
[stderr]    main(_get_train_opts())
[stderr]  File "/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
[stderr]    return f(*args, **kwargs)
[stderr]  File "/tmp/code/mega_nerf/train.py", line 24, in main
[stderr]    Runner(hparams).train()
[stderr]  File "/tmp/code/mega_nerf/runner.py", line 226, in train
[stderr]    dataset.load_chunk()
[stderr]  File "/tmp/code/mega_nerf/datasets/filesystem_dataset.py", line 80, in load_chunk
[stderr]    chosen, self._loaded_rgbs, self._loaded_rays, self._loaded_image_indices = self._chunk_future.result()
[stderr]  File "/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/concurrent/futures/_base.py", line 438, in result
[stderr]    return self.__get_result()
[stderr]  File "/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
[stderr]    raise self._exception
[stderr]  File "/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/concurrent/futures/thread.py", line 52, in run
[stderr]    result = self.fn(*self.args, **self.kwargs)
[stderr]  File "/tmp/code/mega_nerf/datasets/filesystem_dataset.py", line 111, in _load_chunk_inner
[stderr]    loaded_pixel_indices = torch.IntTensor(np.load(self._ray_arrays[next_index]))
[stderr]  File "/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/site-packages/numpy/lib/npyio.py", line 440, in load
[stderr]    return format.read_array(fid, allow_pickle=allow_pickle,
[stderr]  File "/azureml-envs/azureml_42fe9568fca52a4c68567983a86d6e0f/lib/python3.9/site-packages/numpy/lib/format.py", line 787, in read_array
[stderr]    array.shape = shape
[stderr]ValueError: cannot reshape array of size 42028165 into shape (42107792,)

What a great work! May I ask a few questions? If I just want to run a demo, all I need to do is downloading one of the datasets and the pretrained model corresponding to the dataset, and running the create_octree.py to extract the octree and then see the result by the Mega-NeRF-Dynamic viewer, is that correct? Moreover, what's the meaning of --container_path argument and what should I fill in to it? Many thanks!

Hi, thanks for the excellent work.

I want to test the code on a relatively large-scale forward-facing dataset. How to change the code, particularly how to produce the centroids? My understanding is that the parameter ray_altitude_range no longer needs. But I cannot get reasonable cluster mask.

hi, thks for this great work for large scene. i uesd it for my own dataset (500 pics), after training 500k, i get the img from your 'eval_result_rgbs' from the scritpt 'eval.py'. but the result rgb img is not clear enough, do you have any suggestion? any reponse is helpful. thanks!

@hturki thanks for your great work! I'm following the tutorial to see its outcome. However, after the evaluation, I can only see the matrixs.txt, which I want is to see some rendered images or videos. The question is I don't really understand how to creat 3 files of the input param in render_image.py, could you pls offer some examples?

Dear @jaharkes @hturki @therealsatya @teiszler

Thanks the great work! But I have some understing. How to assign a pixel to certain 3D cell? Using the depth of the pixel? There are many sample locations on a ray of a pixel. How to process the sample locations?

Best, Yingjie CAI

This work is excellent. But I have a question, in the paper, you said each submodule can be trained separately, but what if a training ray corresponds to multiple submodules, do you merge the sample points in different modules?

Hi @hturki ! I was trying to compile mega-nerf-viewer but was stuck in the final stage. Do you have any idea about this problem?

OS: Ubuntu 20.04 gcc version: 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1) cmake: 3.24.1 libtorch libx11-dev_1.6.9-2ubuntu1.3_amd64 cuda: 11.6

I have successfully compiled mega-nerf-viewer. Then I ran the following commands with one NVIDIA Tesla A100: ./mega-nerf-viewer ../../OCTREE_PATH.npz --model_path ../../building-pixsfm-8.pt Thanks!