Hi there!

NDF works impressively well on mugs, bowls and bottles. So I'm curious about how it will perform on categories with more complex shapes such as airplanes or chairs.

According to your docs, what I need for training an NDF are generated point clouds and ground truth occupancy values. I've successfully produced the point clouds via https://github.com/anthonysimeonov/ndf_robot/blob/master/src/ndf_robot/data_gen/shapenet_pcd_gen.py. However, it seems that the code to produce ground truth occupancy values is not yet available. Is it possible that you could provide the code? Or maybe could you tell me the procedure to produce the occupancy values?

Thanks a lot!!!

I visualize the demo data and find the shelves intersect with bowls and bottles? Are they supposed to be like this?

Is the demonstration in the teaser available? I think the rack in the teaser is more complicate than the rack in the simulation demos.

Hi, thanks for sharing this great and interesting work!

I'm a bit curious about how partial is the input observed point cloud, as I understand from the paper, the translation equivariance is achieved through subtracting the centre of mass, but this really depends on how complete the point cloud is. If it's too partial, the centre of mass will largely shift from the actual object centre. Since the translation and rotation equivariance are always coupled, from the vector neuron perspective, it is learning the representation with centre shift augmentation and might lead to rotation equivariance error.

Thanks!

I estimated the occupancy of the object point cloud in the demo data, and got very small values like this.

I also estimated with full object points or with dense grid coordinates (instead of 1500 n_pts used above). The results were weird too. I just forwarded the model with centered object points. Did I miss any steps? I am using the multi_category_weights.pth now, and I am curious about the reconstruction performance of the pretrained nets.

I did not find the "pybullet-planning/pybullet_tools/ikfast/franka_panda". Where is the "pybullet-planning/pybullet_tools/ikfast/franka_panda". Thank you!

In the paper it mentioned training on pybullet rendered dataset for simulation experiments. But how about the real world? Seems like it requires non-trivial amount of data to train (100,000 objects for training for simulation experiments). How is this dataset collected in real world to train? Or is it sim-to-real (any quantitative analysis on the sim-to-real gap influence)?

Hello @anthonysimeonov,

I am sorry to disturb you but in line 241 of file src/ndf_robot/model/vnn_occupancy_net_pointnet_dgcnn.py I do not understand to what correspond the T dimension of p. For my understanding p is a batch of 3D points and therefore we have Batch_size=batch size, T=?, D=3 Thank you, Julien.

Hi @anthonysimeonov , Congrats for your work, I really appreciate the idea.

I was wondering whether the descriptors could be used for a different but related task. I am working on 3D assembly (see the breaking-bad dataset for a visual explanation), which means trying to assemble two (or more) broken parts of an object. The idea behind using NDF is that grasping is similar to assembly, because we have two complementary part (the grasp from the robot or the broken part of the object). It is also equi-variant to rotation, which is great for assembling (if you rotate all pieces solution is still ok).

So I was playing around with the code to do some experiments, I can get the latent vector and forward it to get the descriptors, but I am unsure how exactly to use them. Instead of sampling one random 'batch' (many points close to each other outside the surface of the mesh) I am sampling around the whole object and creating the descriptors, and now I would want to match the descriptors from one broken object with the descriptors from the other part of the broken object.

Intuitively, I would expect the descriptors to be complementary (also not sure how to mathematically define complementary in this case), but I see in section II (Method) of the paper, Equation 11 says that minimizing the difference between descriptors is the way to get the transformation. But trying to find correspondences in a "standard" way (i was using example from TEASER++ registration) did not work.

In order to understand better, I was trying to investigate further the concept of energy landscapes, which looks very promising. Can you point me to some code (even part of this repo) to watch to get a better understanding of it?

So, a couple of questiosn:

• if I have a descriptor on a pointcloud (first broken part) and on a second pointcloud (second broken part, same object) and if they belong to the same point, should the descriptors be the same or the complementary?
• how did you manage to create the visualization of the energy landscapes? Where can I look to manage to get one of my own?
• do you think there is a way to use NDF for assembly?

Thanks a lot in advance

Hi, thanks for releasing the source code of this excellent work!

I am really interested in how your baseline of 2D correspondence from DON performs so poorly on placing task, as described in your paper.

Do you have any plan to release your baseline code with DON?

Thanks a million!

Hi!

In the simulation, you used pybullet's functionality to acquire segmentation, how did you acquire it for the actual device? Please let me know how to acquire only the point cloud of the target object with 4 cameras on a real robot. It would be great if you could publish your method or code. Best regards.

Hello!

I want to use NDF, but instead of using the demonstrations to sample query points, I want to do it by manually specifying a point on the object. How should I approach this? Which file do I need to look at? Any guidance will be appreciated.