Thanks for sharing this amazing work! I had a doubt regarding the complete node graph used as input for Occlusion-aware Motion Estimation Network module.

Unlike DeformingThings4D where the complete object surface is known. In datasets like Deepdeform or for live demo, only front-view is available. In these cases what is the input to the module?

1. Is complete object surface precomputed (maybe by DynamicFusion)?
2. Or only the graph extracted from front-view RGBD Image at frame t_0 is used. All the confidence and visibility scores are computed on this graph and no graph update is made during the motion estimation step?

Congrats on your great work! I am currently trying to re-implement your work. I am faced with a problem: you said in your paper that you generate 3D notion motion from optical flow image. I currently can think up of two ways to do it:

1. Project node position to the optical flow image, and read value at that pixel.
2. Compute the motion of each vertex, and compute node motion by averaging the motion of its nearby vertex.

Which one should I choose?

Best, Haonan

Thanks for your wonderful work. I am trying to reproduce your modification on optical flow model RAFT and stumble upon a choice between the full version and subset version (FlowNet2.0 uses this) so I want to ask which version did you use?

There were issues regarding the high memory usage. I couldn't run the demo myself due to CUDA out-of-memory errors.

The solution is to avoid computing gradients at the forward step. As it was before, the gradients accumulate at each iteration since they are not used. This results in linearly increasing memory usage to store unnecessary tensors.

Is it a random process in down sample & up sample?Or you process by another way ? I am confused for it and did't get useful information in your arXiv paper. And Is it feasible to OPEN SOURCES more information about some details about your code release?

This is a really great project, we try to run the demo in our PC (NVIDIA GeForce RTX 3080 Ti). But we encountered this error, can anyone tell me how to resolve this issue. we have tried setting max_split_size_mb but not worked.

File "E:\projects\OcclusionFusion\model.py", line 89, in forward
  feature7 = self.layer72(feature7, edge_indexes[0])
...
RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 12.00 GiB total capacity; 11.10 GiB already allocated; 0 bytes free; 11.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hello, excuse me! First, thanks for your excellent work. Then I want to enquire if this algorithm can get the volume of the real object directly. I will look forward to your reply. Thanks.

Hi! Thank you for the great work! Your real-time results look amazing!

I have a question about the depth image data. It seems like in all of your reconstructed results, the backgrounds (for example the walls) are all removed. May I ask how you guys do it? I mean, it seems like optical flow can solve part of the problem. But optical flows are generated from color images, right? I assume they are not perfect. For example, if you pick all the points that have flow values u^2+v^2 > 1, there will always be some background pixels included in the masked area. Do you set up a threshold for input depth values so you subtract the background in the very beginning? Or do you remove it after you calculated the optical flow? Or do you build everything in the canonical model anyway, you just not visualize it in the experiment? Or anything else?

In some other cases, the person may not move very drastically, so the optical flow may ignore a large part of the person. Do you run into similar problems? Any idea how I can solve this?

Thanks again.