Hi, My GPU is GTX3080Ti and i try to use contact_graspnet(inferency.py).it's pretty slow. I have seen the other issues #16 and #9. I try to use cuda11.2 and cudnn8.1 with TensorFlow-gpu2.5. But it exits a lot of packages conflict and python package bugs. So, is it possible that you provide a new version yml file for TensorFlow-gpu2.5?

Best regards, xiaolin

Hello. I am an undergraduate student currently using contact-graspnet on a project.

I am using contact-graspnet on a different robot (TIAGo from pal robotics) and the gripper differs from the panda robot. In order to be safe, I move the generated grasps backwards by an offset and then perform a forward motion to get the object between the robot's fingers. You have trained the network with the panda's gripper configuration by using its STL model and also points on the gripper (contact_graspnet/gripper_control_points/panda_gripper_coords.yml). I wonder if retraining the network using gripper coordinates for TIAGo's gripper would improve the generated grasps. I think that one of the things it would improve would be the grasps generated based on the gripper width constraints as the panda gripper is wider than TIAGO's gripper.

Thanks in advance!

Hi, have you encountered an issue where the first call to sess.run() in contact_grasp_estimator.py is slow? I am running the inference example in the readme, and when I time sess.run() the first call takes much longer than subsequent calls:

Run inference 1162.3998165130615
Preprocess pc for inference 0.0007269382476806641
Run inference 0.2754530906677246
Preprocess pc for inference 0.0006759166717529297

I found this thread on what seems to be a similar issue but the simple resolutions have not worked, and I have not tried compiling tensorflow from source yet. I am running on a GTX 3090 with CUDA 11.1, tensorflow-gpu==2.2. Have you encountered this issue before? Thanks for your help.

Hello, Thank you again for your code. I tried to change gripper width in config file but I didn't see any difference in inference. I need to train again to change this feature? also I saw the filter with the gripper oppenings but I didn't find a consistent relation, could you explain me it, please.

Hello, Thank you for your awesome code. I'm trying to implement your code with a different gripper, the fingers are longer but I haven't been able to find where to adjust this. Could you help me, please?

Hi,@MartinSmeyer I have a question: how to convert output grasp from inference.py to the real-world coordinate system? I use the following code to calculate the position from pred_grasps_cam,but the result seem incorrect. So I'm wondering if the matrix from pred_grasps_cam contains position and rotation information about the coordinates relative to the object what it want to grasp, or the coordinates actually in the image, or the coordinates actually in real-world?

def rotationMatrixToEulerAngles(R) :

    if R[1,0]>0.998:
        x= 0
        y=math.pi/2
        z=math.atan2(R[0, 2], R[2, 2])
    elif R[1, 0] <-0.998:
        x=0
        y=-math.pi / 2
        z = math.atan2(R[0, 2], R[2, 2])
    else:
        x = math.atan2(-R[1,2],R[1,1])
        y = math.asin(R[1,0])
        z = math.atan2(-R[2,0],R[0,0])

    return x,y,z

def getRotationAndPosition(transformation):
    assert transformation.shape[0] ==4 and transformation.shape[1] ==4 ,"shape error"
    x = transformation[0,3]
    y= transformation[1,3]
    z = transformation[2,3]
    x_r,y_r,z_r =rotationMatrixToEulerAngles(transformation[0:3,0:3])
    return x, y, z, x_r,y_r,z_r

Hi,

I am running into some problems when transforming the output grasp pose to the pose that the robot needs. May I ask if the output grasp pose is in the actual camera coordinate or some other coordinate system? Thank you so much! @MartinSmeyer

Hi,

I was using default parameters to test the model on 7.npy, and the resulting grasps are sparse as shown in the image below. May I ask if I was doing anything wrong?

Here is the command I used: python contact_graspnet/inference.py --np_path=test_data/7.npy --local_regions --filter_grasps

Checkpoint: checkpoints/scene_test_2048_bs3_hor_sigma_001

Thank you! @MartinSmeyer

Hi,

While trying to generate grasps using the following command:

python contact_graspnet/inference.py \ --np_path=test_data/*.npy \ --local_regions --filter_grasps

the code does not proceed after the following print statement:

Generated 1 grasps for object 1.0 Generated 11 grasps for object 2.0 Generated 33 grasps for object 3.0 Generated 134 grasps for object 4.0 Generated 56 grasps for object 5.0 Generated 92 grasps for object 6.0 Generated 66 grasps for object 7.0 Generated 40 grasps for object 8.0 Generated 48 grasps for object 9.0 Generated 5 grasps for object 10.0 /home/kaykay/contact_graspnet/contact_graspnet/visualization_utils.py:63: MatplotlibDeprecationWarning: You are modifying the state of a globally registered colormap. In future versions, you will not be able to modify a registered colormap in-place. To remove this warning, you can make a copy of the colormap first. cmap = copy.copy(mpl.cm.get_cmap("rainbow")) cmap.set_under(alpha=0.0) Visualizing...takes time

It has been around 2 hours, what is the average expected time for the visualization step and if there is a CUDA memory limit (which I suspect) do we get a warning or an error?

predict_scene_grasps_from_depth_K_and_2d_seg() is missing :

• rgb: taken by extract_point_clouds()
• forward_passes: taken by predict_scene_grasps()

Also, extract_point_clouds() does not take local_regions and filter_grasps.

Hi, @MartinSmeyer

I wanted to test the inference.py on real-time data from realsense depth camera. and I realized depth image from the provided test data is normalized while the depth image from realsense camera is uint16 format. I tried to normalize using this formula but i am not certain about the range. depth_normalized = (depth - min_depth) / (max_depth - min_depth) so, could you provide the range used to normalize or any other ways to preprocess depth image?

Thanks, Amanuel

I found you released the issue whether the mmdet can use 4 channel-input? I have achieved this in YOLOv5, and I am also interested in the question you asked? Have you tried in the mmdet?

When I changed the use_farthest_point in the configuration file to true, the training time was very long. It took about 3 seconds to train a scene before it was changed, and it took about 180 seconds to train a scene after it was changed. I also monitored the usage of gpu and found that it was basically useless.

I have noticed that the model produces scores when segmentation mask is set to None. Do these scores have any significance and if so what is that?

P.S. I added your model as a servicecall in a ROSBRIDGE, and, currently, I am naively selecting the top 5 poses with the highest score values without a segmentation mask. Also, i added the the appropriate transforms to get it working on the interbotix wx250s. its surprisingly good even before training and just filtering out invalid gripper widths. I am a senior in undergrad btw, so my journey has been similar to: https://github.com/NVlabs/contact_graspnet/issues/8

I have a problem. When I use the model I trained to predict, the success rate of grabbing is very low, about 0.003. I hope you can give me some help to solve this problem.

Thank you for making the code public. I'm trying this method. But I found that the provided model is in model.ckpt format, and some .meta files are missing, which makes my conversion to '.pb'/'.trt' format always fail, and there are certain problems. Excuse me, can you provide the trained model in '.pb' format, or methods in other formats. I want to try an experimental deployment.

Thank you for the codebase!

Should this line

pcreader.change_object(cad_path, cad_scale)

be changed to

pcreader._renderer._load_object(cad_path, cad_scale)

because SceneRenderer does not implement change_object function.

Thanks!