Hello, I have just discovered your project. how could i use it to do inference on a real time feed?

i am trying to feed a depth image or a point cloud to "some algorithm" that will put them together and store them in a ply. i think this is called point cloud registration? but it overlaps with sfM, where i would take N camera shots and then store a point cloud, then do N+M shots and compare with the previous point cloud to see how much did the new M steps contribute to adding "relevant" points to the PC. or some variant of that maybe with this project.

@hehefan

Can you please explain what does transfer of points to nearest frames mean? how are they transferred? (I understand those anchor points are picked using Farthest point sampling)

Thanks in Advance.

Do I need preprocessing when running the MSR dataset, or do I directly access his depth file? If preprocessing is required, can you provide preprocessing code

Hi, I'm just curious if it's possible to add color information in the form of RGB channels, or even other point-wise features, as input to the model. And, if it is, what kind of modifications would be required on the model architecture. Is it possible/logical to just add extra channels so that instead of passing xyz tensors, we could feed the model the stacked xyzrgb tensors.

Thanks.

To address issue #11 where the pointnet2 installation imports error out, we updated the pointnet2_utils.py file for both Pytorch 1.4.0 & 1.8.1 modules to import from the correct module.

After downloading this repository, I follow readme.md to run python setup.py, but the error occurs:

My current cuda version is 11.6 , and the pytorch version is 1.11.3. I wonder if the above problem can be fixed by just installing the pytorch with the version of 1.8.1. under a new virtual conda environment. Looking forward to your helps, thanks a lot!

hi, i am wondering why you cut one frame into two pieces and use the loss of each piece to update the model when training with synthia4D? Is there any reason to do this? Specifically, i don't understand why using the function called half_crop_w_context in datasets.synthia.

I use the pre-processor from 3DV work, but seems that it will transform the data into .npy files instead of .npz files, so can you provide the code for pre-process of NTU RGBD? Much appreciated.

Hi, I try to run the code and get the result as a baseline. I didn't change the parameters and set the clip len=24, batch size=14, but I only got 89.55 acc for training 100 epochs, while the paper said it should be 90.94. I just don't know why the results are different. I upload my training process, hope you can give me some possible reasons. out_P4.txt

I want to visualize tansformer's attention. I see that Fig4 in your paper visualizes it. Can you tell me where and how to visualize it? Can you share the visualization code? Thank you

Dear @hehefan ,

From the train-msr.py file, I see the default values input points = 2048, ball query radius = 0.7, nsamples = 32, spatial-stride = 32

On a given point cloud frame that contains 2048 points, the farthest points sampled (FPS) i.e., 32 are sampled (smaller). Around each of them, a ball with a radius of 0.7 (bigger) is used to query 32 points (very small number).

I have a few questions to understand the setting.

1. Wouldn't the queried points (32) around the farthest point be very close to it since the overall object contains 2048 points(dense)?
2. What is the intuition behind choosing a bigger radius but querying only 32 points at each FPS point?

I suspect due to 2048 input points, but querying only 32 points around each of 32 farthest points, the points considered from a given frame at a time are limited to 32x32 i.e., 1024 points. And these points are clusters of 32 points around each of the 32 farthest points due to ball querying.

Please help me understand the intuition behind the design.

Thanks in advance.