python main.py --config example_configs/config_vkitti2_Scene06.txt

Loading poses from: ./data/vkitti2/Scene06/clone/pose.txt Loading bbox from: ./data/vkitti2/Scene06/clone/bbox.txt Loading info from: ./data/vkitti2/Scene06/clone/info.txt /home/j316chuck/dev/nerf/neural-scene-graphs/data_loader/load_vkitti.py:308: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray bboxes = np.array(bboxes)[selected_range] Loaded vkitti (6, 375, 1242, 3) [375, 1242, 725.0087] ./data/vkitti2/Scene06/clone [5.] in this scene [6.] in this scene [12.] in this scene [4.] in this scene [9.] in this scene [10.] in this scene 2022-07-05 01:43:27.951776: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2022-07-05 01:43:27.981609: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-07-05 01:43:27.981708: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.8 pciBusID: 0000:01:00.0 2022-07-05 01:43:27.982037: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2022-07-05 01:43:27.983233: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2022-07-05 01:43:27.984156: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2022-07-05 01:43:27.985026: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2022-07-05 01:43:27.986967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2022-07-05 01:43:27.987981: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2022-07-05 01:43:27.991330: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2022-07-05 01:43:27.991421: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-07-05 01:43:27.991541: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-07-05 01:43:27.991592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2022-07-05 01:43:27.992007: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA 2022-07-05 01:43:28.016309: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz 2022-07-05 01:43:28.016706: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x382e820 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2022-07-05 01:43:28.016828: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2022-07-05 01:43:28.088636: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-07-05 01:43:28.088792: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x38014c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2022-07-05 01:43:28.088806: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 3090, Compute Capability 8.6 2022-07-05 01:43:28.088915: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-07-05 01:43:28.088981: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.8 pciBusID: 0000:01:00.0 2022-07-05 01:43:28.089010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2022-07-05 01:43:28.089017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2022-07-05 01:43:28.089022: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2022-07-05 01:43:28.089028: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2022-07-05 01:43:28.089033: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2022-07-05 01:43:28.089039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2022-07-05 01:43:28.089044: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2022-07-05 01:43:28.089071: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-07-05 01:43:28.089137: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-07-05 01:43:28.089184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2022-07-05 01:43:28.089205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2022-07-05 01:43:28.089330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-07-05 01:43:28.089336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2022-07-05 01:43:28.089342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2022-07-05 01:43:28.089382: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-07-05 01:43:28.089452: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-07-05 01:43:28.089505: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 2022-07-05 01:43:28.089523: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21873 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6) MODEL 63 27 <class 'int'> <class 'int'> True (?, 90) (?, 63) (?, 27) MODEL 319 54 <class 'int'> <class 'int'> True (?, 373) (?, 319) (?, 54) MODEL 319 54 <class 'int'> <class 'int'> True (?, 373) (?, 319) (?, 54) Found ckpts [] get rays done, concats adding object nodes to each ray (2794500, 15, 3) Using Ray Object Node intersections Removing object None WARNING:tensorflow:From /home/j316chuck/dev/nerf/neural-scene-graphs/neural_scene_graph_helper.py:510: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where 5 is hit by 15285.0 rays! This is 6 times less than the most hit object! Adding Car support Factor 1.0 /home/j316chuck/dev/nerf/neural-scene-graphs/prepare_input_helper.py:437: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray _eq_sz_rays = np.repeat(np.concatenate(np.array(_new_rays_rgb[_id_hit]), axis=0), _hit_factor, axis=0) 6 is hit by 57126.0 rays! This is 1 times less than the most hit object! Adding Car support Factor 1.0 12 is hit by 5044.0 rays! This is 18 times less than the most hit object! Adding Car support Factor 1.0 4 is hit by 4320.0 rays! This is 22 times less than the most hit object! Adding Car support Factor 1.0 9 is hit by 13278.0 rays! This is 7 times less than the most hit object! Adding Truck and Van support Factor 13.0 10 is hit by 95678.0 rays! This is 1 times less than the most hit object! Adding Car support Factor 1.0 Adding dense sampling close to objects. (3074703, 15, 3) shuffle rays done Begin TRAIN views are [0 1 2 3 4 5] TEST views are [1, 5] VAL views are [1, 5] WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

• https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
• https://github.com/tensorflow/addons
• https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue.

2022-07-05 01:46:12.119972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2022-07-05 01:46:40.414762: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED Traceback (most recent call last): File "main.py", line 1859, in train() File "main.py", line 1738, in train verbose=i < 10, retraw=True, **render_kwargs_train) File "main.py", line 696, in render all_ret = batchify_rays(rays, chunk, **kwargs) File "main.py", line 595, in batchify_rays ret = render_rays(rays_flat[i:i+chunk], **kwargs) File "main.py", line 450, in render_rays raw_bckg = network_query_fn(pts, viewdirs, network_fn) File "main.py", line 945, in network_query_fn netchunk=args.netchunk) File "main.py", line 60, in run_network outputs_flat = batchify(fn, netchunk)(embedded) File "main.py", line 27, in ret return tf.concat([fn(inputs[i:i+chunk]) for i in range(0, inputs.shape[0], chunk)], 0) File "main.py", line 27, in return tf.concat([fn(inputs[i:i+chunk]) for i in range(0, inputs.shape[0], chunk)], 0) File "/home/j316chuck/anaconda3/envs/neural_scene_graphs/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 898, in call outputs = self.call(cast_inputs, *args, **kwargs) File "/home/j316chuck/anaconda3/envs/neural_scene_graphs/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 695, in call return self._run_internal_graph(inputs, training=training, mask=mask) File "/home/j316chuck/anaconda3/envs/neural_scene_graphs/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 844, in _run_internal_graph output_tensors = layer(computed_tensors, **kwargs) File "/home/j316chuck/anaconda3/envs/neural_scene_graphs/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 898, in call outputs = self.call(cast_inputs, *args, **kwargs) File "/home/j316chuck/anaconda3/envs/neural_scene_graphs/lib/python3.6/site-packages/tensorflow_core/python/keras/layers/core.py", line 1050, in call outputs = gen_math_ops.mat_mul(inputs, self.kernel) File "/home/j316chuck/anaconda3/envs/neural_scene_graphs/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6126, in mat_mul _six.raise_from(_core._status_to_exception(e.code, message), None) File "", line 3, in raise_from tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(49152, 63), b.shape=(63, 256), m=49152, n=256, k=63 [Op:MatMul]

With certain train/test splits on Scenes 1 and 2 (using the settings in https://github.com/princeton-computational-imaging/neural-scene-graphs/issues/7) I see either a black background or nothing at all:

Any guidance on what could be causing this would be greatly appreciated!

hi, great work! During training, I find that for several scenes such as scene 3,4,7, their PSNR can only achieve about 13. If use pretrained examples_6, the PSNR of Scene6 can achieve about 21(still lower than 23 in the paper) I wonder what is the reason? Are there any training tricks? Below is my conifg of Scene7, which is modefied from examples of Scene6: config_kitti_0007_example_train.txt

Looking forward to your reply Thanks

Thanks for your great work and sharing your code to reconstruct the dynamic scene.

I have seen the paper mentioning that the NSG can learn the shadow implicitly, and saw there is use_shadows in the arguments. However, this argument seems not to be used anywhere in the code base. Is this because the release version did not have such functionality?

Best, Kaicheng

The code here shows that the training set contains all frames in the scene, and the validation (test) set is the part of the training set. Is this setting reasonable for a novel view synthesis task?

Hi, I really appreciate your amazing work.

After reading your code and demo, I found there are still some traces in demo, objects are rendered in a box style.

I also check your code, you have tested the instance segmentation in early experiments. I am really interested in the reason why you choose bounding box instead of instance segmentations.

Best

Hi, I have trained a model on Virtual KITTI 2, following the training pipeline from readme. However, when I use the trained model for inference, I set "render_only" to True and only get three frames of inference results. So I would like to ask you how I can get a long video like the one shown in your project page based on this code. Looking forward to your reply, thank you!

Thanks for the paper and making your code public. I had some questions around reproducing the results in Table 1 of the paper:

• To confirm, is the training sequence is some set of input frames from a KITTI sequence (frames 65 to 120 for Scene 06) and then are the evaluation numbers reported by rendering all of the training set frames (ie: is the validation set identical to the training set?) or some subset
• Is the evaluation script present in this repo? Running python main.py --config example_configs/config_kitti_0006_example_render.txt seems to render 3 frames + a video of those 3 frames, but I'm not sure that it's showing any evaluation metrics.
• Are the numbers reported against subsequences of scenes 0001, 0002, and 0006? If so, would it be possible to learn the starting and ending frames used for 0001 and 0002? The README mentions config files for those scenes but I don't see them in the repo

"PackagesNotFoundError: The following packages are not available from current channels:

• imagemagick" There is no windows version of imagemagic in conda channels "https://conda.anaconda.org/conda-forge/" I can find Win64 install package in website http://www.imagemagick.org/script/download.php but how to use it in the project？

Hi,

I run the example in the Rendering a Sequence with 'bckg_only=True'. Also, I test the code for all dynamic parts with 'obj_only=True' & 'white_bkgd=True'. However, both two cases have a error. ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 112 and the array at index 1 has size [98.]

This is because in the main.py line 1518. rays_rgb = np.concatenate([rays, images[:, None, ...]], 1) The size of rays is (112, 2, 375, 1242, 3) and the size of images is (98, 1, 375, 1242, 3). They can not concatenate together. How to solve this problem?

Thank you for any suggestion. Letian

Hello! Follow your steps to install the environment and relevant data. The experimental equipment and environment meet the requirements. According to the pre training model, only three steps are performed to generate a 500K vido that cannot be opened mp4。 Prompt the following warning， It's where I didn't set it？ BoZhang